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An evaluation was conducted of a 3-year demonstration 
program the purpose of which was to demonstrate the efficacy of 
projects that included vocational education as a key component in 
encouraging at-risk youth to remain in or return to school. Six of 
the 10 project grantees, funded under the Cooperative Demonstration 
Program of the Carl D. Perkins Vocational Education Act, were 
evaluated, using process and outcome information from 12 localities. 
The study found that only one-third of the projects achieved 
significant reduction in the number of participants who dropped out 
of school. Projects were generally more successful, however, in 
improving participants' school performance and affiliation, with 10 
of the 12 demonstrating increases in participants' grade point 
average, 7 showing reduction in number of courses failed, and 7 
showing improvements in participants' perceptions of the safety of 
their school environment. Among the components that appeared to be 
critical for improving the persistence and educational success of 
at-risk youth are the following: (1) a smaller, more personal 
environment, such as that available in the school-wi thin-a-schoo 1 and 
alternative school environments; (2) a structured environment that 
includes clear and equitably enforced behavioral expectations; (3) 
vocational education that integrates academic education and leads to 
good entry-level jobs or continued training at the postsecondary 
level; (4) formal, ongoing coordination of the academic and 
vocational components of participants' high school programs; (5) a 
formal counseling component that incorporates attention to personal 
issues along with career counseling, employabi 1 i ty and life skills 
instruction; and (6) personal, supportive attention from adults 
through mentoring or similar projects. (The report includes separate 

evaluations of each site, 13 tables, and 3 appendixes that report on 
the study methodology.) (KC) 
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EXECUTIVE SUMMARY 



In February 1989, the Office of Vocational and Adult Education (OVAE) in the U.S. 
Department of Education (ED) initiated a three-year demonstration program whose purpose 
was to demonstrate the efficacy of projects that included vocational education as a key 
component in encouraging at-risk youth to remain in or return to school. Authorized under 
the Cooperative Demonstration Program (CDP) of the Carl D. Perkins Vocational Education 
Act (P.L. 98-524, Section 411), the demonstration targeted grants to communities with high 
dropout rates. OVAE awarded 10 grants under the demonstration authority to school districts, 
universities, and other types of organizations that proposed to replicate a tested dropout 
prevention model, expand services already available in the locality, or develop a new design 
for improving the school persistence of target youth. As part of its overall effort in this 
demonstration, ED also awarded a contract for a rigorous evaluation of the grantees. This 
document summarizes the findings of the evaluation's final report. Included are sections on 
the evaluation's purposes and methods, findings, and implications. 

Purposes and Methods 

The evaluation's design included assessment of the dropout prevention projects' 
impacts on participant outcomes, including attendance, school performance, students' attitudes 
and aspirations, and persistence to graduation; and examination of project implementation 
over the three-year period of the grants. Six of the 10 CDP grantees participated in the 
evaluation; since some were multisite projects, the evaluation collected process and outcome 
information from a total of 12 localities. 

Specific purposes of the evaluation were as follows: 

• To determine the effectiveness of the projects in reducing dropping out and 
other at-risk behaviors and attitudes associated with dropping out; 
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• To analyze student outcomes as a function of what services the projects 
actually provided over the period of the grant (as opposed to what they 
described in their grant proposals); 

• To investigate factors that affected project implementation, in order to facilitate 
decisionmaking regarding adoption or adaptation of particular dropout 
prevention/reentry project models in other locations for specific types of youth; 
and 

• To examine study design-related decisions and experiences throughout the 
course of the study to ensure the most rigorous evaluation possible. 

To address the evaluation purposes having to do with participant outcomes, the study 
team implemented one of two designs, based on the nature of the projects selected for 
participation in the evaluation and other factors: (1) random assignment of members of the 
projects' applicant pools to treatment or control groups, and (2) a comparison group design 
that matched participants to nonparticipants on key characteristics. In addition, we 
implemented a supplementary evaluation that used a gap reduction design. The gap reduction 
technique involves selection of a nonequivalent group of "typical" students for comparison 
with study participants to determine the extent to which receipt of services reduces the gap in 
performance between average students and those with low achievement, attendance, and other 
behaviors that are characteristic of the at-risk students targeted by the projects. 

Of the 12 project sites participating in the evaluation, nine implemented random 
assignment, while the remaining three implemented a matched comparison group design. 
Students in 10 of these sites were represented in Cohort 1, which received services starting in 
the 1989-90 school year; all 12 sites were represented in Cohort 2, which received services 
starting in the 1990-1991 school year. The two cohorts were also combined into a single 
cohort for analytic purposes. 

There were 1,062 students included in Cohort 1, and 1,430 students in Cohort 2. 
Approximately 27 percent of the students participating in the evaluation were in treatment 
groups (i.e., received dropout prevention/reentry services from one of the projects). Thirty- 
two percent of the participants served as statistical comparison groups to the participants in 
the treatment groups. The remainder were in the gap reduction groups. Nontreatment 
condition students were assigned to either a randomly assigned experimental control group or 

ii 
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a quasi-experimental comparison group, with group membership determined by the research 
design selected by the dropout prevention/reentry site. 

To address evaluation purposes having to do with identification of the actual 
"treatment" received by project participants, we implemented a concurrent longitudinal 
process evaluation, or implementation study. This component of the evaluation included 
periodic visits to each participating site that included onsite observations of project activities, 
collection of information from records, and interviews with administrators, teachers, other 
staff involved in delivering services, and participants. Depending on each project site's actual 
implementation schedule (which ranged from fall 1989 to fall 1990), study staff visited 
projects twice a year for two years and again at the end of the three-year demonstration 
period. 

The evaluation design required collection of substantial amounts of quantitative and 
qualitative information from a variety of sources, including (1) project applicants, participants, 
and members of the control and comparison groups; (2) student and school records; (3) 
administrators of projects' host organizations; (4) persons involved in project administration; 
(5) instructors and others involved in delivering the projects' services; and (6) members of the 
private sector. Implementation of the random assignment and comparison group designs 
required collection of background information, including student characteristics and 
educational experiences and status. Comparison of educational (e.g., grade point average) and 
attitudinal (e.g., perceptions of teachers and counselors) outcomes of participants and 
control/comparison groups required collection of information from records and through 
student and staff interviews over the life of the evaluation. Development of causal inferences 
about the influences of the interventions on outcomes required collection of baseline 
information on each project's intended services and ongoing measurement of the extent to 
which projects modified their service designs over time. Several of the grantees found it 
necessary to make fairly extensive modifications to their intended service designs; the 
implications of these changes for participant experiences and outcomes were an important 
aspect of the study's ongoing data collection and analysis activities. 
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Study Findings 



The CDP demonstration's primary intended outcome was reduction in dropping out 
among project participants, or, for reentry projects, return to school. Given the likelihood that 
projects might achieve other, intermediate, outcomes (e.g., improvement in school affiliation 
or performance) prior to changes in dropping out, we also examined a number of these 
outcomes. In sum, the number of the 12 total project sites that achieved each of the 
outcomes of interest, relative to the outcomes of the comparison or control group, in the 
evaluation were as follows. 



Number of Projects With Outcome 

4 



Outcome 

• Reduction in dropping out 

• Increase in grade point average 10 

• Reduction in number of courses failed 7 

• Increase in number of credits earned 5 

• Reduction in number of absences 5 

• Improvement in students' perception of 

teachers and instruction 4 

• Improvement in students' perception of 

counselors and counseling 2 

• Increase in students' perception that 

school is safe 7 

• Students' perception of receiving more academic 
encouragement 4 

• Students' perception of receiving better job 
preparation 3 

As this summary notes, only one-third of the projects achieved significant reduction in 

the number of participants who dropped out of school. Projects were generally more 
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successful, however, in improving participants' school performance and affiliation, with 10 of 
the 12 demonstrating increases in participants' grade point average, seven showing reduction 
in number of courses failed, and seven showing improvements in participants' perceptions of 
the safety of their school environment. Five projects achieved reduction in the number of 
absences. 

In terms of project implementation, and its relationship to outcomes, we found the 
following. 

• Organizational location, in combination with the location of project sites and 
services, affected project implementation and participant outcomes in that 
projects with close proximity to the grantee organization tended to fare better 
in terms of these outcomes. 

• Grantees experienced more success in implementing project models with which 
they were familiar. 

• Project success depended on careful analysis of the types of youth targeted for 
services, and on provision of services that were appropriate to targeted youth. 

• Some level of integration, or at least coordination, of academic and vocational 
course work appeared important in engaging students and leading to 
improvement in their overall performance. 

• All projects, even those that did not originally intend to do so, implemented 
some form of counseling or mentoring support; most projects with such 
services showed improvements in students' affiliation with school. 

• While the study suggests that participation in vocational education did have 
salutary effects on student performance, the variability in what the projects 
delivered as "vocational education" suggests the need for further study of the 
efficacy of vocational education in reducing dropping out. 

• Projects that changed the structure of the school, particularly those that created 
a smaller, more nurturing, environment achieved positive effects in school 
performance and affiliation, though not in retention. 
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Components of Effective Dropout Preve n tion FrogranM in Vocati .>nal Rduratinn 



Review of the experiences of the projects funded under the CDP demonstration from a 
summative perspective permits reflection on those components of the projects, taken together, 
that appear to have promise for improving the persistence and educational success of at-risk 
youth. Among the components that appear critical are the following: 

• A smaller, more personal euvu-onment, such as that available in the school- 
within-a-school and alternative school environments; 

• A structured environment that includes clear and equitably enforced behavioral 
expectations; 

• Vocational education, preferably that contains integration of academics with the 
vocational content, and, for most participants, has an occupational concentration 
leading to good entry-level jobs or continued training at the postsecondary 
level; 

• Formal, ongoing coordination of the academic and vocational components of 
participants' high school progranfis; 

• A formal counseling component that incorporates attention to personal issues 
along with career counseling, employability development, and life skills 
instruction; and 

• Personal, supportive attention from adults, through a mentoring or other project 
component. 



CHAPTER 1: INTRODUCTION 



In February 1989, the Office of Vocational and Adult Education (OVAE) in the U.S. 
Department of Education (ED) initiated a three-year demonstration program. Its purpose was 
to demonstrate the efficacy of various strategies that included vocational education as a key 
component in encouraging at-risk youth to remain in or return to school. Authorized under 
the Cooperative Demonstration Program (CDP) of the Carl D. Perkins Vocational Education 
Act (P.: . 98-52^, Section 411), the demonstration provided for awarding of grants to school 
districts and other eligible entities located in communities with high dropout rates. Grantees 
were to (1) replicate project models found to be effective in other settings (e.g., the high 
school academy model; Project COFFEE), (2) expand an existing project that met the 
objectives of the demonstration, or (3) develop new designs to meet locally identified needs. 
Because a key intent of the program was that projects be capable of widespread replication in 
other settings, grantees were to incorporate formal dissemination activities into their 
operations. Among the strategies suggested for implementation were: 

• Coordination of remedial academics or general education development (GED) 
preparation with vocational training; 

• Implementation of cooperative programs with the private sector; 

• Conduct of incentive-based programs that would reward schools for reducing 
their dropout rates; 

Implementation of flexible scheduling that would permit dropouts to combined 
employment with continuing education. 

In July 1989, OVAE awarded 10 grants under the demonstration authority. Table 1-1 
shows the types of organizations that received these grants: four of the recipients were local 
education agencies, two were regional education centers within a state, and one was a state 
board for vocational education. Other grantees were a community college, a university, and a 
local branch of a national charitable organization. 

Several grantees planned to implement their project in multiple sites; in all, the 10 
grantees operated projects in a total of 16 locations. These locations included comprehensive 



Table 1-1 

Types of CDP Grantee Organizations 







i-tucai eaucanon agency 


4 


Community college 


1 


University 


1 


Regional education service center 


2 


State vocational education board 


1 


Loca] charitable organization 


1 


TOTAL 


10 



high schools, area vocational-technical centers, K-12 schools. Bureau of Indian Affairs (BIA) 
schools, and alternative schools, one located in a store front and the other in an old storage 
building that the school's faculty and students renovated). Because of the late award of the 
grants, some projects were unable to begin operations as expected in the fall of the 1989-90 
school year Several started in the spring semester of that year, while a few got underway the 
following fall. The projects continued through the end of the 1991-1992 school year. 

Evaluation of the Demonstration Prelects 

As part of the demonstration program, ED decided to conduct a rigorous evaluation of 
the demonstration projects' implementation and effects. ED's Planning and Evaluation 
Service (PES) awarded a contract for this evaluation to RMC Research Corporation (with 
subcontracts to the American Institutes for Research and Research Triangle Institute) in 
February 1989. This report contains findings from that evaluation, which was conducted over 
the life of the demonstration program. The evaluation's overall purpose was to assess. 
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through implementation of a rigorous experimental design, the extent to which projects with 
vocational education as a key component improve the educational experiences and outcomes 
of at-risk youth. Six of the 10 grantees, representing a total of 12 of the 16 project sites, 
were selected for the evaluation, based on the suitability of their service design to a rigorous 

longitudinal evaluation. 

Included in the study's design were (1) evaluation of projects' impacts on participant 
outcomes, including attendance, school performance, students' attitudes and aspirations, and 
persistence to graduation, and (2) an intensive examination of project implementation over the 
three-year period of the grants. Specific purposes of the evaluation were as follows: 

To determine the effectiveness of the projects in reducing dropping out and 
other at-risk behaviors and attitudes associated with dropping out; 

To analyze student outcomes as a function of what services the projects 
actually provided over the period of the grAit (as opposed to what they 
described in their grant proposals); 

To investigate factors that affected project implementation, in order to facilitate 
decisionmaking regarding adoption or adaptation of particular dropout 
prevention/reentry project models in other locations for specific types of 
students; and 

To examine study design-related decisions and experiences throughout the 
course of the study to ensure the most rigorous evaluation possible. 

To address the evaluation purposes having to do with participant outcomes, the study 

team implemented one of two designs, with selection based on the nature of the projects 

selected for participation in the evaluation and other factors (e.g., two school districts had 

policies against random assignment and hence were unable to implement the design of 

choice). These designs v/ere (1) random assignment of members of the projects' applicant 

pools to treatment or control groups and (2) a comparison group design that matched 

participants i nonparticipants on key characteristics. In addition, a supplementary evaluation 

using the gap reduction design was implemented. The gap reduction technique involves 

selection of a nonequivalent group of "typical" students for comparison with study 

participants to determine the extent to which receipt of services reduces the gap in 
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performance between average students and those with low achievement, attendance, and other 
behaviors characteristic of the at-risk students targeted by the projects. 

To address evaluation purposes having to do with identification of the actual 
"treatment" received by project participants, we implemented a concurrent longitudinal 
process evaluation, or implementation study. This component of the evaluation included 
periodic visits to each participating site that included onsite observations of project activities, 
collection of information from records, and interviews with administrators, teachers, other 
staff involved in delivering services, and participants. Depending on each project site's actual 
implementation schedule (which ranged from fall 1989 to fall 1990), study staff visited 
projects twice a year for two years and one additional time at the end of the three-year 
demonstration period. 

The evaluation design required collection of substantial amounts of quantitative and 
qualitative information from a variety of sources, including (1) project applicants, participants, 
and members of the control and comparison groups; (2) student and school records; (3) 
administrators of projects' host organizations; (4) persons involved in project administration; 
(5) instructors and others involved in delivering the projects' services; and (6) members of the 
private sector. Implementation of the random assignment and comparison group designs 
required collection of background information, including student characteristics and 
educational experiences and status. Comparison of educational (e.g., grade point average) and 
attitudinal (e.g., perceptions of teachers and counselors) outcomes of participants and 
control/comparison groups required collection of information from records and through 
student and staff interviews over the life of the evaluation. Development of causal inferences 
about the influences of the interventions on outcomes required collection of baseline 
information on each project's intended services and ongoing measurement of the extent to 
which projects modified their service designs over time. Several of the grantees found it 
necessary to make fairly extensive modifications to their intended service designs; the 
implications of these changes for participant experiences and outcomes were an important 
aspect of the study's ongoing data collection and analysis activities. 
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Organization of This Report 



This report presents the findings from the implementation and outcome components of 
the evaluation. It is organized into five chapters. Chapter 2 describes the methodology we 
employed for both the implementation and the outcome analyses. Chapter 3 presents findings 
on the projects' implementation during the life of the demonstration. Chapter 4 presents the 
results of the outcome analyses, considered in light of the projects' implementation status 
over time. Finally, Chapter 5 discusses the implications of the evaluation's findings for 
policymakers and practitioners concerned with improving educational and supportive services 
and outcomes for at-risk youth. 
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CHAPTER 2: METHODOLOGY 



The evaluation's longitudinal design included two interrelated components. The first 
component was a longitudinal implementation, or process, study, involving periodic site visits 
over a three-year period to each of the 12 project sites in the in-depth evaluation. The other 
component was analysis of the impacts of these projects on participants through collection of 
information from individuals in the treatment and control/comparison groups and from student 
records (e.g., attendance, grades, credits earned). This chapter provides an overview of the 
methodology associated with each of these components. 



Process Evaluation 

The process evaluation had two major purposes. The first purpose was to obtain 
information on services actually provided to participants in order to help explain the types of 
outcomes experienced by project paiticipants. The second goal was to analyze factors that 
affect project implementation in order to facilitate decisionmaking on adoption or adaptation 
of particular models of dropout prevention/reentry projects in other locations, given the needs 
of the target population, characteristics and environments of localities, and monetary and other 
resources available to implement strategies for encouraging at-risk youth to remain in or 
return to school. 

The design of the process evaluation was longitudinal, with twice-yearly visits to each 
of the projects in the in-depth study in the first two years of project operations and a final 
visit during the spring of the third year. Activities conducted during these visits included the 
following: collection of records information on school and community context (such as 
district organization and community demographic characteristics), project staff and monetary 
resources, and project-related documents developed by the grantees; observation of project 
operations through classroom visits and visits to other relevant locations (e.g., worksites, 
where appropriate); and interviews with local administrators and key project staff (director, 
teachers, counselors). 
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We developed a set of instruments for conducting these activities, which permitted 
monitoring of changes over time in project goals, activities, staffing patterns, and the like. 
These instruments included the following: 

• Implementation Checklist 

Tailored to the project model undertaken by each of the grantees in the in- 
depth evaluation, the checklist guided site visitors in rating the level and 
quality of implementation at the project component level (e.g., attendance 
monitoring, work experience). 

• Project Resource Record 

This instrument collected information on financial, staff, and other resources 
used by the project, including types and amounts of financial contributions (in 
addition to the federal grant), volunteered time, and nonfinancial contributions 
(e.g., computers), in support of project activities. 

Staff Training and Experience Record 

Administered to all staff, this instrument collected information on the 
educational background and experience of all staff, including experience in 
working with at-risk youth, participation in staff development activities related 
to the project, and extent of contacts with project participants, parents, and 
others in support of project activities. 

• Classroom Observation Form 

This form provided a framework for observing classroom activities, including 
types of instructional methods employed, interactions between instructors and 
participants, and general environment of the classroom. 

• School/Community Context Record 

Site visitors used this form to collect basic contextual information relevant to 
the project, including dropout rates, overall demographic characteristics of the 
community, and district profile (size, organization) information. 

• District/School Administrator Interview Guide 

This protocol guided interviews with district or school administrators 
responsible for activities relevant to the project, such as local initiatives for 
improving educational services to at-risk students, support for the project. 
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factors associated with the project's implementation, and plans for continuation 
of the project following the demonstration period. 



Project Director Interview Guide 

This instrument guided detailed interviews with project directors and explored 
factors that have affected project implementation, changes in project objectives 
or activities, resource and management issues, perceptions of the effectiveness 
of the project, and factors relevant to project replication. 

Project Teacher Interview Guide 

This protocol was used for collecting information on the project's instructional 
components, including teacher perception;^ concerning the appropriateness of 
the design, the model's strengths and weaknesses, issues that arose over time i 
implementing the model, and effects on participants. 

Ancillary Staff Interview Guide 

This protocol guided interviews with counselors, social workers, and other 
support staff concerning their role in the project, the tailoring of services to 
individual students, and their perceptions of the eificacy of the project m 
meeting the ancillary needs (e.g., personal support) of participants. 

Private Sector Interview Guide 



This protocol permitted exploration of contributions of private sector 
representatives to the project, including provision of mentors and internships, 
donation of equipment or supplies, and other activities designed to assist 
projects in meeting their objectives. 
Site visits to the projects for the process evaluation component of the study began 
during fall 1989, and were scheduled to accommodate the initiation dates of each of the 12 
sites in the in-depth study.' Chapter 3 presents findings from the process evaluation, and 
Chapter 4 integrates information from the process and outcome evaluation components to 
address the relationship between project activities and implementation status and participant 



outcomes. 



'The dropout prevention/reentry projects in Detroit and Broward County were not implemented 
1990-1991 academic year, and several others began in spring 1990. 
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Outcome Evaluation 



The outcome evaluation was designed to obtain information over the course of the 
study on participant outcomes and changes in outcomes (comparing treatment to comparison 
groups) such as attitudes toward self and school, educational and employment aspirations, 
school performance, attendance, dropout rates, and employability. The design called for 
analysis of these outcomes for each of the "models" implemented in the demonstration (e.g.. 
Career Academies). As a group the grantees that participated in the evaluation did plan to 
implement several discrete models (e.g., two Career Academy programs, seven replications of 
Project COFFEE), the projects that they actually implemented differed substantially from site 
to site. Consequently, we presem the evaluation's process and outcome findings by site, 
rather than by project model or for the demonstration as a whole. 

The outcomes reported in Chapter 4 of this report were based on data collected from 
the demonstration projects at 12 sites. Students in 10 of these sites were represented in 
Cohort I, which received services starting in the 1989-90 school year; all 12 sites were 
represented in Cohort 2, which received services starting in the 1990-1991 school year. The 
two cohorts were also combined into a single cohort for analytic purposes. 

A total of 1,062 students were included in Cohort 1, and a total of 1,430 students were 
included in Cohon 2. Approximately 27 percent of the students participating in the 
evaluation were in treatment groups (i.e., received dropout prevention/reentry services from 
one of the projects). Thirty-two percent of the participants served as statistical comparison 
groups to the participants in the treatment groups. Table 2-1 presents the sample sizes for the 
participating projects by experimental condition (i.e., treatment and control/comparison 
group). Nontreatment condition students were assigned to either a randomly assigned 
experimental control group or a quasi-experimental comparison group, with group 
membership determined by the research design selected by the dropout prevention/reentry 
site.'^ 



'Visits to each dropout prevention/reentry site were completed during fall 1989 to encouraee the 
development of an experimental research design. Nevertheless, several projects implemented other design''s v^ith 
nonequivalent quasi-cxpcriinenfal comparison groups. 
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Table 2-1 



to 

I 



Dropout Prevention and Reentry Projects in Vocational Education Sample Sizes, by Site and Cohort 




* Projects not implemented until 1990-1991 academic year 
** Cohorts without gap reduction groups 
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Additionally, we implemented a supplemental research design, called gap reduction,^ 
in all project sites to protect against the degradation of the experimental and quasi- 
experimental designs that might result from differential attrition. Approximately 41 percent 
of the study participants were in "Typical Student Groups" used for the gap reduction 
analyses. These students were not directly involved in the study. Rather, they served as a 
comparison group used to assess the reduction in the size of the gap between treatment 
students and "typical" students from before to after treatment on key variables of interest. 
Table 2-2 displays the research design employed by the projects. 

Table 2-2 

Research Designs Employed by Dropout Prevention and Reentry Projects, by Site 

Research Design | 



Project Site 


Random 


Matched 


Gap Reduction 


Assignment 


Comparison 


Design 


Woodside 




^ 

• 




Carlmont 




• 





Gushing 



Oconee 



Ann Arundel 



Broward 



Portland 



Detroit 



Turtle Mountain 
Fort Totten 
Fort Berthold 
Fort Yates 



^Tallmadge, G.K., Lam, T.C.M., & Gamel, N.N. (1987). Bilin/^ual education evnluatinn and reporting 
system: Users' guide. Washington, DC: U.S. Department of Education. 
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The findings presented in this report are based on data collected at three or four time 
points: before project initiation (i.e., the baseline), in the 1989-90 school year, in the 1990-91 
school year, and in the 1991-92 school year. In other words, the 1991-92 outcome data 
reflect the status of students after three years of dropout prevention services for Cohort 1. 
For Cohort 2, the 1991-92 data represented two full years participation in the project. 

Measures 

As for the implementation study, we developed a set of instruments for collecting 
outcome data such as dropout status, school performance, affiliation, and attitudes. The 
instruments included the following: 

• Initial Student Questionnaire 

This instrument collected baseline information on student educational, 
attitudinal, and employment outcomes, including attitude toward school, 
aspirations, and labor force entry, and collected demographic and other 
background information not available elsewhere. 

• Follow-Up Student Questionnaire 

This instrument was used to collect mid-treatment and post-treatment measures 
of student educational, attitudinal, and employment outcomes, including attitude 
toward school, aspirations, and labor force entry. It was also used to update 
information on students' living arrangements and other variables subject to 
change over the duration of the study. 

• Student Exit Interview Protocol 

This protocol was administered to students who dropped out of the program or 
school, transferred to another school, or completed the program or high school. 
It collected post-treatment measures of student outcomes, obtained students' 
evaluations and perceptions of the project, and identified the students' reasons 
for leaving school or the project. 

• Student Semester Record Form 

This form was employed once each semester to collect information on student 
outcomes, including attendance, grade point average, standardized test scores, 
and exit status. 



This section describes refinements of the measures and analytical methods for the 
analyses of data collected in the 1990-91 and 1991-92 school years. 

Dropout Status. Some students who drop out of school later return to continue their 
studies. To provide an objective criterion measure that would not be subject to the bias of 
censoring in a finite study, dropout status was computed at particular dates: June 30, 1991, 
for the 1990-91 school year, or June 30, 1992, for the 1991-92 school year. In other words, 
students who dropped out of school in March 1991 but returned to school at the start of the 
next school year would be counted as a dropout in analyses of the status at the end of the 
1990-91 school year but might be counted as a continuing student in the later analysis.* 

Because there was variation in the completeness with which schools keep track of 
nonattending students,' dropouts were defined in three separate ways for analysis: 
(1) individuals formally classified as dropouts, (2) both dropouts and individuals formally 
expelled, and (3) dropouts, expulsions, and individuals classified as "moved," with no 
additional information about graduation or transfer into another school While all 
conceptualizations of dropout status were considered during data analyj s, this report employs 
the second conception of dropping out, which we believe more completely represents the 
behavior of student dropouts. 

Absences. Students are absent from school for a variety of reasons (e.g., unexcused 
absences, illness, approved school field trips); however, school records typically do not 
differentiate among them. A dropout prevention program might be expected to have an effect 
on unexcused absences, but it would not have an effect on other types of absences, which 
may vary from one year to the next. We conducted a series of analyses to identify a 
transformation of absences that would remove chance variation, leaving the reliable variation 
that is measured by a within-group correlation from one year to the next. As a result, the 
number of days absent in a semester was adjusted in two ways: (1) all absences in excess of 



''students in the treatment group who exited the dropout prevention/reentry program without dropping out 
of school were continued as program participants for analytic purposes. 

^In many cases, the reporting of dropout status was delayed by several months or the project reports were 
incomplete (e.g., gave only the semester of the drop). 
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27 days were ignored, to avoid assigning excessive weight to a small number of individuals 
with very high absence rates; and (2) absences were "clumped" into four or five day 
categories (e.g., fewer than five days absent was treated as a single outcome value, from five 
to eight days was treated as another value, and from nine to 13 days as another value). 

Demogra phic groups . Although the comparison groups were designed to be matched 
to the treatment groups, the match was imperfect, and differences in attrition between 
treatment and comparison groups increased the discrepancy. To eliminate this variation from 
the comparisons between groups, all treatment/control comparisons were conducted within 
demographic categories and the results were pooled across the categories to obtain an overall 
result for each site. 

Three factors were used to match the groups: gender, race/ethnicity (i.e., white/non- 
white), and relative age (old for grade/age appropriate). The definition of relative age was 
based on a comparison of the student's birthdate and current grade level with the typical 
birthdate of students in the same grade in the same school year. A student was categorized as 
overage when his or her birthday was more than one full year eariier than the typical 
birthdate. 

The analyses were carried out in such a way that over-representation of one group in 
the treatment, compared to the control group, would not bias the results. However, in 
carrying out treatment/control comparisons, it became necessary to eliminate a small number 
of cases with data due to the unavailability of comparable individuals in the other group. In 
other words, if the control group at one program site included no older, nonwhite females, 
then the older, nonwhite females in the treatment group could not be included in the 
comparison. Thirty-five students in Cohort 1 and 37 students in Cohort 2 were deleted for 
lack of a match, although this procedure changed slightly from variable to variable. Tables 
A.l - A.3 in Appendix A present sample sizes for each variable by program site, experimental 

condition, and cohort. 

Factor analysis of the attitudinal data from the Initial Student Questionnaire was 
conducted to identify item clusters that could form scales and decrease the number of separate 
■ variables during analysis. We intercorrelated all of the attitudinal items on the questionnaire 
and performed a principal components factor analysis on the resulting correlation matrix. 
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Twelve factors were extracted that had eigenvalues greater than unity. Following a varimax 
rotation, we found five identifiable factors with at least three items with high factor loadings 
(above .55). The scales and the items they included follow. 



• Attitude toward teachers/teaching 

--Students got along well with teachers. 

—The teaching was good. 

-Teachers were interested in students. 

-When I worked hard on my school work, my teachers praised my effort. 
-In class, I often felt "put down" by my teachers. 
-Most of my teachers really listened to what I had to say. 
-Most of the courses I took were interesting and challenging. 

• Expectations for the future 

-Do you think that graduation from high school helps people get better jobs? 
-Have you decided what you will do after you leave high school? 
-Which best describes your expectations for the future? 

• Attitude toward counselors/counseling 

-A guidance counselor helped me develop a four-year educational plan 

for high school. 
-A guidance counselor helped me develop a career plan. 
-I feel that my high school prepared me to look for and obtain 

appropriate employment. 

• Perception of academic encouragement received 

-I was encouraged to take more English courses. 

-I was encouraged to take more math and science courses. 

-I was encouraged to enroll in more difficult math and science courses. 

• Sensitivity to classmates' disruptive behavior 

-Other students often disrupted class. 

-Disruptions by other students got in the way of my learning. 
-Misbehaving students often got away with it. 

Seven other constructs were treated as single-item factors. In all, five scales and 10 

additional attitudinal items were analyzed separately. There were also four school records 
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items: grade-point averages, number of days absent, number of courses failed, and number of 
credits earned. 

In some cases, incomplete data were available for some study participants. To 
facilitate the evaluation of nonresponse bias resulting from the missing data, all analyses were 
conducted both on the original data and on data with missing information imputed using 
procedures developed by study researchers. The data imputation procedures used linear 
relationships between all existing data points to develop linear predictors of each variable.* 
Frequency distributions of the existing data for each variable were then developed separately 
for each site, and values from the distributions were sampled to fill in missing data points.^ 
After careful consideration, all analyses in this report used raw (unimputed) data. Given the 
relatively large levels of missing data to be imputed in some cases (e.g., > 25 percent of the 
information, we believe the analyses with unimputed data provide more defensible and 
generalizable findings. 

Analytical Methods 

The purpose of the analyses was to identify differences between the treatment and 
control/comparison groups and between the treatment and gap reduction groups at each site. 
There was no attempt to match the characteristics of the gap reduction group to the treatment 
group. Comparisons between the experimental groups were conducted using a variety of data 
analytic approaches to address different threats to the validity of the resulting inferences. 

Analyses of dropouts . The most important outcome for the study was the student's 
continuation in school. Thus, we examined whether more control group students dropped out 
of school than treatment group students. To address this question, which involves a 



'The linear prediction equations were based on the data combined across project sites. Treatment or control 
group membership was not considered during the imputation process. Not including group membership in the 
prediction equations ensured that, other characteristics equal, a response would be selected from the same distribution 
whether the student was in the treatment or control group. Given the differences in data collection approaches and 
population characteristics, imputation of missing data with participants in the Gap Reduction Comparison groups was 
not performed. 

'For students missing multiple responses, the imputations were sequential, using the relations computed in 
the first step, with the imputed value for the first variable included in the imputation of later variables. 
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dichotomous outcome (dropped out or continued in school), we conducted Mantei-Haenszel 
analyses.^ The Mantei-Haenszel test is an extension of the X^-test, used when data are 
subclassified by factors controlled in the analyses. Within each demographic category of 
student, a two-by-two table (treatment-control by dropout-nondropout) was constructed, and 
these tables were combined across demographic categories at a site (i.e., gender, 
race/ethnicity, and relative age) to estimate the aggregate difference between the observed and 
expected number of treatment group students who dropped out.' The relative rate of 
dropping out and the relative odds of dropping out were computed as summary statistics, and 
the probability of obtaining as large as or larger difference by chance was computed. Both 
the one-tailed probability, on either tail, and the probability of obtaining an outcome with a 
smaller probability, were computed using statistical procedures developed by the researchers. 

Analyses of other outcomes . Dropout prevention programs may have effects on a 
variety of other outcomes, ranging from grades and absences to attitudes about school. Each 
of these outcomes can be considered as an approximation, to be measured on an interval 
scale. In that case, it is appropriate to model the effects of the demonstration program as a 
linear model; and the research question (for each measure) becomes: "Does exposure to the 
treatment contribute significantly to the variance of the outcome measure?" 

In setting up a linear model, three background variables (gender, race/ethnicity, and 
relative age) were dichotomized and treated as "class" variables, so that the effect of the 
treatment could be construed as the pooled within-class effect of the treatment and therefore 
be unconfounded by treatment-control differences in these background characteristics. 

In order to sharpen the analysis, the potential outcome measures were also measured 
prior to exposure to the program so that each student's outcomes could be compared to his or 
her starting point. Although the treatment and control groups were initially matched at each 
site, this would also help to remove bias that might arise from differential attrition (i.e., if 
control group members with extremely high absences dropped out and were lost from the 



^Mantel. N., & Haenszel. W. (1959). Statistical aspects of the analysis of data from retrospective studies 
of dihease. Journal of the National Cancer Institute. 22, 719-748. 

'The expectation assumes that treatment and control group students drop out of school with equal likelihood. 
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analysis but treatment group members with high absence rates were retained, then the analysis 
would be biased against finding an effect). Therefore, all linear model analyses of outcomes 
took pretreatment measures into account. 

The method of preference for these analyses is gain score analysis; that is, prediction 
of the post-measure minus the pre-measure, based on group membership. However, to 
provide a picture of the extent to which conclusions are robust with respect to method of 
analysis, three other methods of analysis were also used. First, to account for pre/post 
variance differences, a probabilistic gain score analysis was substituted in some cases for 
simple gain score analysis. (A detailed discussion of all analytical methods including their 
derivations is presented in Appendix C of this report.) Second, analyses of covariance were 
carried out, using the pre-measure as a covariate in an analysis of variance in the post- 
measure. The weakness of this analytical method lies in its reliance on the assumption that 
the pre-measure is an error-free measure of the true score component of the outcome variable, 
a highly unlikely assumption. 

Strictly speaking, regular analysis of covariance is appropriate only for randomized 
control group evaluations where the random equivalence of the treatment and control groups 
has not been destroyed by differential attrition. Where the groups are not randomly 
equivalent, regular analysis of covariance will usually underadjust for pretest differences 
between groups. Lord (1960) and Porter (1967) have suggested reliability-corrected 
covariance analysis as an alternative for adjusting for pretreatment differences between 
groups that are not randomly equivalent. These analyses take into account the error in the 
covariate; however, they depend on a precise estimate of the reliability of the pre-measure of 
the outcome. Because an independent estimate of the reliability was not available, the 
reliability-corrected covariance analyses were somewhat problematic in that we had to 
estimate the reliabilities of the variables we used. The problem was in obtaining a stable 
estimate of the pre/post correlation. Since the sample sizes at most individual sites were too 
small to ensure stable estimates of the correlations (some were even negative), the 
correlations (within-site) were pooled across sites to obtain a single estimate of the reliability 
of each pre-measure. Thus, the reliability adjusted analysis of covariance used pooled within- 
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group pretest-posttest correlations (none of which was smaller than +0.239). Under an 
assumption of equal variance between pre-measures and post-measures, this is mathematically 
equivalent to gain score analysis. 

Reliability adjusted covariance analyses will, presumably, not underadjust for 
pretreatment differences in nonequivalent comparison group quasi-experiments, but it will 
usually overadjust for pretreatment differences in randomized experiments. We made an 
informed decision to use both forms of covariance analysis for all of our sites, reasoning that 
two estimates of treatment effect sizes would serve to bracket the correct value. 

The regular analysis of covariance results would be underadjusted for all sites since 
some of them were evaluated using a nonequivalent comparison group design and even the 
randomized control group evaluations were compromised by nonrandom attrition. The 
reliability-corrected analysis of covariance results would be overadjusted for all sites for the 
same reasons. We felt that the two analyses, considered together, would be more informative 
than either by itself. 

Adoption of the gap reduction design was originally intended as a backup to the other 
designs (e.g., to be used when random assignment was corrupted or when quasi-experimental 
comparison groups experienced high levels of differential attrition). As it turned out, 
however, the design has some unique advantages for some of the individual-site evaluations. 
In one site, for example, all members of both the treatment and control groups spend half of 
each day in a vocational-technical school. The treatment is an add-on to that half day, but the 
possible impact of the vocational-technical school itself is a matter of some interest. Our gap 
reduction comparison group was drawn from the population of students who attended the 
"sending" schools all day. For this reason, the gap reduction design offered us the possibility 
of assessing the impact of the vocational-technical school in addition to assessing the impact 
of the dropout prevention project. 

Analyzing the differences between treatment and control/comparison groups in 
multiple ways yields different effect-size estimates for each analytic approach. For example, 
we first employed regular analysis of covariance (presumably the correct form of analysis for 
randomized control group designs where the random equivalence of the groups has not been 
destroyed by differential attrition). Next, we used a reliability-corrected covariance analysis, 
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appropriate where the treatment and control groups are not randomly equivalent.'" Finally, 
we implemented the gap reduction evaluation design, which assessed the reduction in the size 
of the gap between the treatment groups and a group of "average" students drawn from the 
same schools as those attended by study participants, from before to after treatment." The 
results of all of these analyses were expressed as "effect sizes" (mean post-treatment 
difference between groups divided by the control/comparison group's standard deviation or 
the standardized amount of gap reduction) with confidence intervals. 

The results of the outcome analyses are presented and discussed in Chapter 4. 



^°See, for example, the following: Lord, F.M. (1960). Large-sample covariance analysis when the control 
variable is fallible. Journal of the American Statistical Association, 55, 307-321. Porter, A.C. (1967). The effects 
of using fallible variables in the analysis of covariance. Unpublished doctoral dissertation. University of Wisconsin, 
Madison. 

^^Tallmadge, G.K., Lam, T.C.M., & Gamel, N.N. (1987). Bilingual education evaluation and reporting 
system: Users' guiup. Washington, DC: U.S. Department of Education. 
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CHAPTERS: IMPLEMENTATION 



This chapter presents findings from the study's process evaluation, based on periodic 
site visits to the projects that participated in the evaluation. As noted in the previous chapter, 
our data sources for the implementation study included documents provided by each site, 
interviews with project and, as appropriate, district and school officials and teachers, 
observations, and discussions with students. Included in the chapter are: (1) an overview of 
the grants funded through the Office of Vocational and Adult Education's (OVAE) 
Cooperative Demonstration Program (CDF), the program authority for the demonstration; (2) 
findings on the implementation of projects participating in the in-depth evaluation; and (3) 
implications of those findings for project effects. The information reported in this chapter 
then forms the basis for discussion of the relationships between project processes and 
outcomes contained in Chapter 4 of this report. 



Overview of the Cooperative Demonstration Program 



To be eligible for funding under CDP, grantees were required to make vocational 
education a key component of their service design, implement that design in a locality with a 
high dropout rate, and select a design that is capable of widespread replication. Grantees 
were invited to test a variety of strategies for encouraging at-risk youth to remain in or return 
to school. Examples of such strategies are: 

• Coordination of remedial academic or GED instruction with vocational training; 

• Implementation of cooperative projects with the private sector; 

• Conduct of incentive-based projects that would reward schools that reduced their 
dropout rates; 

• Use of flexible scheduling that would permit dropouts to combine employment with 
continuing education. 
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Projects were authorized to replicate a dropout prevention model that had been found 
effective in other settings, adapt a locally developed model, or expand a project currently in 
operation in their locality. 

In all, OVAE awarded 10 grants to school districts, state agencies, and other 
organizations that proposed to test a variety of strategies to meet the purposes of CDP. These 
organizations received funds to implement projects over an 18-month period beginning in 
school year 1989-90, with continuation of the grants for an additional 18-month period 
contingent on acceptable progress during the initial funding period. While all projects were 
continued for the full three years, some were funded at slightly lower levels during the 
continuation period. Table 3-1 shows the organizations funded under the program, their 
locations, and the amount of funds awarded for the three years of the demonstration. As 
shown in the table, grants ranged in size from a low of $60,000 for 18 months to a high of 
just over $1 million. Total funding for the demonstration was $8.6 million, with two grantees 
receiving over $1 million and one under $350,000 for the three-year period. 

A specific criterion for award of a demonstration grant was that the project be 
implemented in an area with a high dropout rate. As shown in Table 3-2, nearly all 
participating sites reported high rates, ranging from a cohort rate of over 50 percent in 
participating schools located on or near two of the North Dakota reservations to a low of 12 
percent for one of the reservations'. Nearly all grantees reported rates that amounted to at 
least 25 percent for a cohort, and several reported twice that rate. 

Four of the grantees were local school districts or cooperatives of districts, while two 
were postsecondary institutions, one a state board of vocational education, two were 
educational services organizations, and one a private not-for-profit charitable organization. 
Some were located in metropolitan areas (e.g., Detroit, Baltimore, Fort Lauderdale), while 
others operated in rural locations (e.g.. Gushing, Oklahoma; Seneca, South Carolina; four 
Indian reservations in North Dakota). One postsecondary institution worked with three school 
districts to implement projects in local schools. 



'One participating school reported a dropout rate of 22. ."i percent; one, a rate ot'O percent; the third did not report 
a rate. 
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Table 3-1 



Funding Levels of CDP Grants 



; Grantee 




:\ 










Initkd 


Coitduuutmn y 


*^ 1 ^^^^ !r« 


Sequoia Union HS District* 


Redwood City, CA 




$ 410.313 


$ 439.506 


$ 849.819 


jl Rancho Santiago College 


Santa Ana, CA 




$ 387,528 


$ 289,715 


$ 667.243 


j Baltimore Co. Public Schools 


Baltimore. MD 


$ 359,245 




^ /I o.yuj 


1 Detroit Public Schools* 


Detroit. MI 




■4> 1 iKJ\j I yyy\i 


S; 503 893 


$1,505,889 


]j Southern Westchester BOCES 


Valhalla. NY 




$ 373.847 


$ 410.988 


$ 784,835 


\\ North Dakota Voc. Ed. Board* 


Bismarck, ND 




$ 459.408 


$ 690,573 


$1,149,981 


;i Central Area Voc Tech School* 


Cushing. OK 




$ 342.536 


$ 414,616 


$ 757.152 


ji 

'i Multnomah Co. District No. 1* 


Portland. OR 




$ 448.935 


$ 416,309 


$ 865.244 


II 

;j Clemson University NDPC* 


Clemson, SC 




$ 395.961 


$ 528.563 


$ 924.524 


ll 

Ij Anne Arundel Co. Schools 
I; Broward Co. Schools 
l' Oconee Co. Schools 


Annapolis. MD 
Ft Lauderdale, FL 
Seneca. SC 




[60.000] 
[60.000] 
[60,000] 


[60.000] 
[60.000] 
[60.000] 


[120,000] 
[120.000] 
[120.000] 


'! ' ' ' 

!' Catholic Charities of Richmond 


1 Richmond. VA 




$ 143,017 


$ 194.177 


$ 337.194 


1 

TOTAL 




$4,322,786 


$4,248,000 


$8,570,786 



'Grantees participating in the evaluation 
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Table 3-2 



Dropout Rates in Grantees' Localities 







Sequoia Union High School District 
Redwood City, California 


60% minorities* 
40% statewide minorities* 
30% statewide all* 


Rancho Santiago College 
Santa Ana, California 


50%* 


j Baltimore County Public Schools 
! Baltimore, Maryland 


6% overall' 
14% target schools' 


1 

i School District of the City of Detroit 
1 Detroit, Michigan 


42%* 


,1 Southern Westchester BOCES 
■j Valhalla, New York 

,i 


9.6% overall' , 
23% target area* 


;i North Dakota State Board for Vocational Education 
1 Bismarck, North Dakota 

j Fort Yates 
Fort Totten 
! Turtle Mountain 
1 Fort Berthold 


j 

For participating | 
schools 

58%* i 
51%* i 
17%* ! 

12%* i 

i 


|| 

! Central Area Vocational Technical School 

'1 v^Uaning, v/KlolluiUa 


25%* 

1 


1 Multnomah County District No. 1 
; Portland, Oregon 


8.2%' 

! 


■ Clemson University National Dropout Prevention Center 
Clemson, South Carolina 

1 

il Anne Arundel County Schools, Maryland 

Broward County Schools, Florida 
J Oconee County Schools, South Carolina 


1 25%* 
1 41%* 

1 35%* 


'1 Catholic Charities of Richmond 
^ Richmond, Virginia 


! 4.3% overall' 
j 28% target youth' 


"Cohort rate 
'Annual rate 
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Several of the grantees implemented projects in more than one school or location. As 
noted previously, the Clemson University project edited a validated dropout prevention 
model in three local districts. Additionally, the North Dakota State Board for Vocational 
Education planned to implement the sanae noodel at schools located on or near four Indian 
reservations in the state. Finally, the Sequoia Union High School District replicated a model 
in two high schools located within that district. In all, then, the 10 grantees operated projects 
in 16 separate sites. 

Overview of Protects in the Evaluation 

As noted in Table 3-1, six of the 10 grantees were selected for the in-depth evaluation. 
Criteria upon which their selection was based included (1) the project's plans to provide 
services to a sufficiently large number of par iCipants, (2) the likelihood that the project's 
service design would be of sufficient length and intensity to afford the potential for 
measurable outcomes,^ (3) suitability for one of the designs planned for the evaluation, and 
(4) agreement to participate in the study. These six grantees yielded a total of 12 project 
sites, 10 of which were sufficiently far along during school year 1989-90 to participate in the 
first-year's data collection activities. The other two initiated evaluation activities in fall 1990. 

Table 3-3 provides an overview of the characteristics of the project sites along several 
dimensions. As shown, nearly all of the projects served students who had not yet dropped 
out, while one served mainly dropouts working toward a GFD. Nine were replicating an 
existing model. One was adapting a locally developed project initially designed for students 
with limited English proficiency. Another project was adapting, for at-risk youth, a locally 
developed model initially targeted to students with learning disabilities. The other was 
expanding an existing project by using the grant to add new components. Seven of the 
projects intended to operate as alternative schools. Services planned by the 12 project sites 



^Two of the projects implemented an open entry/open exit or crisis intervention design, which meant that 
many panicipuits might receive specific services for very short periods. Thus measurement of outcomes of 
interest to the study (e.g., changes in attendance or grades) would not be possible. 
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Table 3-3 



Characteristics of Projects in the In-depth Evaluation 







— ' -■■■■■-'.1 


Grade levels served 






9 - 12 




11 


Reentry 




1 



I Purpose of the grant 



Replicate a validated model 9 

Project COFFEE (7 sites) 

High school Academy (2 sites) 
Adapt a locally developed project 2 
Expand an existing project 1 



Organizational structure 

Alternative school 7 

School-within-a-school 2 

Supplemental services 3 



Key services 

Formal vocational education 

Career awareness/employability devt 

Paid work experience 

Academic remediation 

Academic enrichment 

Counseling 

Incentives 



6 
6 
5 
7 
3 
4 
9 



included formal vocational education, with several projects planning to support students in 
their regular vocational programs; career awareness, work experience, academic components, 
and counseling. 
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The Project Models 

Recipients of grants under CDP had the option to replicate a validated dropout 
prevention/reentry model, replicate or adapt a locally developed model, or expand a project 
already underway. Two of the grantees-Clemson University and the North Dakota State 
Board for Vocational Education-received awards to replicate Project COFFEE (Cooperative 
Federation for Educational Experiences), a nationally validated and often replicated model. 
This model thus accounted for seven of the 12 project sites included in the in-depth 
evaluation. In brief, COFFEE is an abbreviated-day alternative school that integrates 
academic and vocational instruction to increase the likelihood that participants will complete 
school and be ready to enter the labor force in good entry-level jobs. Targeted toward highly 
at-risk youth, the model's key features include small class sizes, a highly structured and 
nurturing environment, a strong personal counseling component, career awareness counseling, 
student participation in entrepreneurial businesses that generate income, a physical education 
program that stresses recreational and leisure activities rather than competitive sports, and 
intense monitoring and evaluation of student progress. Teachers typically receive release time 
to work individually with students and to plan an integrated educational program for 
participants. 

Another grantee, the Sequoia Union High School District in California, replicated the 
Peninsula Academy model in two of the eight high schools in the district. This district is the 
site of the original two Peninsula Academies, which have been operating for about 10 years. 
The Academy concept was recently adopted by the California State Department of Education, 
with the legislature making funds available for widespread replication of the model in high 
schools throughout California. 

The Academy model is a school-within-a-school. Beginning in tenth grade, 
participants are block scheduled into most of their classes, including core academic subjects 
and a vocational program (e.g., computer science, health occupations). The model emphasizes 
close partnerships with the private sector, which identifies individuals to serve as mentors to 
students, provides assistance in program design, often donates funds or equipment, and makes 
internships (or work experience positions) available to students. Students who make 
acceptable progress in terms of grades and attendance are rewarded with jobs in the summer 
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following their junior year and in the second semester of senior year. The model features 
reduced class size, incentives to students, and reduction in teachers' classroom assignments to 
facilitate planning and preparation. 

The Gushing, Oklahoma, project adapted a model developed by the grantee (the 
Oklahoma Child Service Demonstration Center) for adolescents with learning disabilities. 
The Oklahoma Developer Demonstrator model, which is validated by the Joint Dissemination 
Review Panel of the National Diffusion Network (NDN), trains prescriptive teachers and other 
school staff to adapt instructional methods and materials for students with special learning 
needs; provides a resource center including a professional staff and computer-managed 
assessment, instruction, and feedback in basic academic (and vocational) skills; and includes 
affective interventions designed to improve students' self-concept and school affiliation. The 
model includes comprehensive inservice training for instructional staff. 

The Portland, Oregon, grant intended to expand the school district's vocational 
continuum through implementation of a Vocational Mentoring program in one of the city's 
high schools. Based on the Experience-Based Career Education (EBCE) model, which also is 
validated by the Joint Dissemination Review Panel of NDN, Vocational Mentoring includes 
the following components: (1) a split-day schedule under which participants attended regular 
high school for one-half day and the vocational program the other half, for mentor-based 
work experience and applied academics; (2) occupational exploration and mentoring 
opportunities in allied health professions, implemented in coordination with a local hospital; 
(3) integration of academic and vocational skills through an adjusted curriculum; (4) life and 
employability skills development activities; and (5) participant incentives. The project 
targeted highly at-risk youth who were experiencing difficulty in the regular high school 
environment and whose academic achievement levels precluded enrollment in regular 
vocational programs available in the district. 

The grantee's schedule for incorporating the Vocational Mentoring component of the 
district's Vocational Continuum precluded initiation of evaluation activities with Vocational 
Mentoring students during the project's first year. Consequently, the evaluation drew a 1989- 
1990 cohort from the existing BRIDGE program, an intervention whose purposes are to help 
middle school students make a successful transition to high school and introduce them to 
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vocational education and employment training options that are available to them in high 
school. The project provides a BRIDGE class each day with a reduced class size. The 
curriculum includes employability development, strategies for improving study and coping 
skills, peer relationships, and other topics such as self-esteem development. 

The Detroit, Michigan, project adapted an existing district-sponsored project (Preparing 
Bilingual Youth for Employment Program) to serve a broader target group of at-risk youth 
(including students with learning disabilities, low achievers, and youth who would otherwise 
be ineligible for enrollment in vocational education because of low grade point averages or 
credit deficiencies). Named STARS (Support Team for At Risk Students), the model is 
designed to permit these at-risk youth to benefit from programs available in the district's five 
Vocational-Technical Centers through providing specialized supportive services, including 
remediation, advocates for limited English proficient youth, special instructors to assist 
students in achieving competency in their vocational courses, and special counselors. 
Coincident with the award of the grant in late summer 1989, the district experienced major 
changes in the composition of the school board as well as a change in district leadership. A 
resuhing district reorganization and hiring freeze meant that the project was unable to initiate 
services during fall or winter of the 1989-90 school year. For this reason, the evaluation was 
unable to begin data collection during the first year of the study, and the site was not 
included in the evaluation's data collection until the second year. 

In the next section of this chapter, we discuss in some detail the projects' goals, 
services, participants, and other characteristics based on our site observations and interviews. 
Of specific interest were challenges faced by the projects in getting underway during school 
year 1989-90, and strategies adopted to meet those challenges. Additionally, the section 
addresses ongoing challenges many projects faced over their three-year life. Later sections of 
the chapter explore the implications of the projects' designs and operations for improving the 
educational experiences and outcomes of their participants. 
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Findings on Project Implementation 



Project Initiation 

As noted previously, projects received their grant awards in summer 1989, after the 
close of school and generally too late for many of them to make the necessary staffing and 
other arrangements necessary to initiate services to participants at the beginning of the 1989- 
90 school year. Consequently, many of the projects needed part or all of that year to 
complete planning, staffing, and other "start-up" work prior to beginning to serve participants 
(Table 3-4). Four of the projects were able to begin serving students in September 1989, 
although none was fully operational. Carlmont and Woodside began serving students but did 



Table 3-4 
Initiation of Project Services 



! Project 


Initiation Date 


; Anne Arundel County, Maryland 


January 1990 




Broward County, Florida 


September 1990 




Carlmont, CA 


September 1989 




Cushing, OK 


September 1989 


1 

■II 


Detroit, Michigan 


September 1990 




Fort Berthold, ND 


March 1990 




Fort Totten, ND 


March 1990 




Fort Yates, ND 


March 1990 


1 

i 


■ Oconee County, SC 


' March 1990 




Portland, OR 


i September 1989 




Turtle Mountain, ND 


March 1990 




Woodside, CA 


September 1989 


I 
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not have the business technology labs set up until late in the first semester. Gushing 
completed most start-up activities in time for the fall semester but did not obtain the software 
and other equipment for the resource room until the middle of that semester. Although most 
of the Portland vocational continuum was already in place, the component funded under the 
grant and selected for the evaluation was not scheduled to begin until fall 1990. Thus, while 
participants and comparison group members were selected and followed during 1989-90, they 
did not enter the "treatment" under evaluation during the project's first year of operations. 

The Clemson project scheduled the fall semester of the first project year for 
development activities, with services to students intended to begin in January 1990. Two of 
the three project sites implemented by Clemson were able to begin serving students during the 
first year: Annapolis in Januar>' as planned and Oconee County in mid-March. The third site, 
Broward County, Florida, did not initiate services during the first year, but started in 
September 1990. Finally, the four project sites established by the North Dakota grant began 
serving students in March 1990. 

A number of factors accounted for these variations in the times that the projects began 
servmg participants. First, because grantees received their awards in the summer, some of the 
grantees needed to identify and assign (or hire) staff. School districts routinely make the next 
year's staff assignments in spring, and making reassignments can be complicated and time 
consuming. North Dakota and Oconee hired project directors in December and February, 
respectively, and each site spent most of first semester and the early weeks of the second 
semester identifying and hiring instructional and other staff members. 

A second issue faced by several projects over the first six months or so was 
arrangement of facilities and equipment. Understandably, this activity was relatively more 
challenging for multiple-site projects like North Dakota and Clemson than for single-site 
locations like Cushing. The COFFEE model calls for implementation of the alternative 
school in a separate facility. Thus, North Dakota had to arrange for facilities and equipment 
in four localities that were widely dispersed around the state. In this project, arrangement of 
space was further complicated because each of the four projects drew participants from 
several schools. Distances were great-at Fort Berthold, for example, the eventual 
arrangement to locate the project in a trailer in New Town meant that students from White 



Shield had to be bused 60 miles each way. Some of the projects (Anne Arundel, Fort Yates, 
and one of the schools participating in the Fort Totten site) eventually located the project in 
one of the local schools. This and other factors meant that the intent to replicate the 
COFFEE model broke down, and a "revised" version was developed. 

The three projects (Gushing, Carlmont, and Woodside) that were able to start in 
September 1989 had made arrangements for facilities prior to the grant award. Gushing, for 
example, had obtained written commitment from the Gentral Area Vocational-Technical 
School (CAVT) in Drumright to locate the project at the area school, including office space 
for instructors and a counselor and use of the resource room as a learning center housing a 
computer lab. Sequoia had reached agreements with two of the district's eight high schools 
to implement a Business Technology Academy should the district win a GDP grant. Even 
with these projects, however, space issues affected some project activities. At Garlmont and 
Woodside, arrangements had to be completed for converting classrooms to secure locations 
for extensive computer equipment; these arrangements, in addition to delivery of the 
equipment, hindered implementation of that project component. Similariy, in Gushing, some 
of the computer hardware and software did not arrive until mid-semester; consequently that 
project service was late in beginning. Further, Gushing had to overcome participants' 
aversion to the computer lab; it was located in what was formeriy a special education 
resource room, which the school's student population viewed as the "dummy's room." It took 
project staff some time to overcome this stigma and make students understand the usefulness 
of the services offered through the lab's instructional and equipment resources (and thus be 
willing to go there). 

The greater ease that these three projects experienced in getting underway, even given 
the av/ard of the grants during the summer, probably resulted from the grantees' experience 
with the particular models they were implementing. The project director of the Sequoia grant 
was principally responsible for implementing two Academies in high schools in the district a 
number of years before, and had continued to be extensively involved with that model at 
district and state levels (and also nationally) for a long time. The Gushing project director 
was involved in developing the model from which the GDP project was adapted and had also 
spent considerable time in recent years assisting other districts around the nation to replicate 
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the project model. These two individuals, along with other staff in their localities, were thus 
highly knowledgeable about the logistical details of initial implementation and were able to 
"hit the ground running" once the grants were awarded. 

The other project that began ser\'ing students in September 1989 was Portland's 
vocational continuum. The Portland project used the grant to incorporate new components 
into their systemwide set of projects and services for at-risk students. As noted earlier, the 
component under study in our evaluation (a Vocational Mentoring project located in an area 
hospital) was not scheduled to begin until fall 1990. The BRIDGE component that we 
investigated during 1989-90 (based on the understanding that many BRIDGE students would 
enroll in Vocational Mentoring in fall 1990) had already been operating in the district for 
about three years; thus the grantee did not need to undertake any project initiation activities 
other than addition of staff to accommodate expansion of services associated with the grant. 
The project then had a full year to finalize arrangements for space, facilities, equipment, and 
staff for the Vocational Mentoring project. 

Conversely, the North Dakota project director was not involved in writing the grant 
proposal. The COFFEE model was selected by state-level decisionmakers who believed that 
it would work well in the intended settings. However, lack of familiarity with the model may 
well have affected the efficiency of early developmental activities. Similarly, the local sites 
that participated in the Clemson replication of COFFEE lacked experience with the model. 
However, in this instance, the grantee incorporated a semester's planning and development 
period, which included extensive training on COFFEE and other activities intended to 
facilitate implementation. 



Project Goals 

Expectably, the overriding goal of all 12 projects was to reduce dropping out among 
at-risk youth. More explicitly, each of the projects had the goal to demonstrate the 
effectiveness of its project model in reducing the dropout rate. Gushing, for example, 
intended to demonstrate the rcplicability of the GRADS project in other rural localities, while 
the Clemson project had the goal to identify the issues that local districts must address in 
replicating or adapting COFFEE in both rural and urban districts. Similarly. North Dakota 
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planned to demonstrate the effectiveness and replicability of COFFEE in preparing highly at- 
risk youth, living in areas of high unemployment, to achieve productive adult lives. Portland 
planned to demonstrate the effectiveness of implementing linked services that would 
constitute a comprehensive continuum of vocational programs and services that would result 
in a significant increase in the numbers of at-risk middle and high school students who would 
both persist in their education to a diploma or GED and be prepared for employment or 
postsecondary enrollment. The Academies projects intended to demonstrate the effectiveness 
and replicability of that model in preparing youth for entry-level skilled and semiskilled 
employment or postsecondary education. 

In response to the needs of their target populations, all of the projects articulated 
intermediate objectives to support their main project and student-focused goals. Recognizing 
that academic deficiencies often discourage youth and lead to dropping out, 10 of the 12 
projects focused specifically on improvement in academic achievement (see Table 3-5). 
Three (Carlmont, Woodside, and Portland) placed particular emphasis on improved self- 
esteem among participants, while four (the North Dakota reservations) focused on 
improvements in life adjustment skills. Three (Anne Arundel, Oconee, and Portland) 
articulated improved attendance and reduced tardiness as an important objective, and Anne 
Arundel and Oconee emphasized reduction in suspensions or other disciplinary actions as 
intermediate objectives. 

Given the emphasis on vocational education as a dropout prevention strategy, of 
particular interest are project objectives that focus on vocational education. Five projects 
specified acquisition of occupational skills as an important intermediate objective. These 
were projects that included formal vocational education programs as a project component. 
Eleven articulated improved employability, or employability skills, as objectives, and one 
listed improved student knowledge about opportunities in nontraditional occupations. Two 
included employer satisfaction with the work and behaviors of project students and completers 
as an objective. 

As this review suggests, overall the projects articulated goals and intermediate 
objectives that attempted to address the multiple needs and interests of youth targeted for 
services. Emphasis on preparation for employment, including acquisition of both occupational 
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and employability skills (capability to identify jobs, prepare resumes, conduct job interviews, 
and exhibit acceptable work behaviors such as timeliness and ability to get along with 
coworkers and supervisors) reflects the recognition that many youth must earn money to stay 
in school. Additionally, the self-esteem associated with succeeding in a job (in the case of 
Carlmont and Woodside, in earning the privilege of having a job through appropriate 
performance in school) is an ancillary benefit that may improve students' affiliation with 
school. Simdlarly, improvement in self-esteem, behavior, and life adjustment are thought to 
increase the likelihood that participants would be able to handle the school environment and 
persist to completion. Finally, the focus of most of the projects on redressing participants' 
academic deficiencies reflects an understanding that a key factor in dropping out is a student's 
perception that he is so far behind— in credits, particularly-that continuing in school may be 
futile. 

Organizational Location and Administration 

As noted earlier, 11 of the 12 project sites used CDP funds to implement new projects, 
while one added new components to an existing service system. The latter project, Portland, 
expanded the size of the BRIDGE program for at-risk ninth and tenth graden> and 
implemented a new Vocational Mentoring component. All of the projects faced different 
implementation-related challenges, which tended to vary according to the organizational 
location of the grant, requirements for establishing viable linkages with schools or districts, 
and the availability of qualified and experienced staff 

North Dakota faced perhaps the most complex organizational and governance 
challenges. State Board for Vocational Education officials, having selected COFFEE as the 
project model most appropriate for meeting the needs of at-risk youth attending schools on or 
near the state's reservations, applied to the State Department of Public Instruction to obtain 
"pilot status" for the project. This status would have allowed them to obtain a waiver such 
that instructors not certified in English or math (but certified as high school teachers in other 
subjects) would be able to teach courses in these subjects that would afford academic credit to 
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participants.-' However, following award, the Board was unable to obtain pilot status from 
the Department of Public Instruction. The result was that the COFFEE model of integrated 
academic and vocational instruction, with students receiving credit toward graduation in this 
restructured curriculum, could not be implemented. As a "fall-back" arrangement, the 
academic instructor hired at each of the four sites provided tutoring in academic subjects 
along with preemployment instruction, while the vocational instructor provided career 
education and other vocationally oriented services (e.g., development of work experience 
opportunities). 

A second challenge faced by this grantee was that some of the project sites did not 
have vocational programs available to students. (The original intention to "mobilize" 
vocational education through a system of vans that would travel from site to site did not 
eventuate, although in the third year of the grant the project did purchase Apticon vocational 
evaluation systems for each site.) For example, only one of the three participating schools at 
Fort Yates had any vocational programs at all. While the BIA school there had industrial arts 
labs, these were not available to students attending the two public schools (Fort Yates and 
Solen) participating in the project at that location.'* (The BIA school subsequently dropped 
participation in the project altogether, an indication of the difficulties the project director 
experienced with administrators at that school from the demonstration's outset.) At Fort 
Totten, where the high schools were served by an area vocational-technical center, the district 
superintendent of one school was hesitant to bus students to the center. He believed that his 
students would experience discrimination from other students; further, many parents were 
hesitant for their children to leave the reservation. This school subsequently ceased sending 
project participants to the vocational-technical center, which meant that those youth had very 
limited access to vocational education programs. 



'The project needed pilot status because of the scarcity of certified English and math teachers in the localities 
in which the project was to be implemented. 

■"Because students can choose any one of the three schools in the area, public school officials were hesitant 
to bus students to the BIA school for vocational education. The fear was that students might decide to transfer 
to the BIA school; for one of the public schools, such transfers would have had serious implications for funding. 
If the school's five COFFEE students had transferred, the school would have lost approximately $15,000 in state 
FTE funds. 
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Finally, the project director, located in the state capital, faced the major challenge of 
developing active cooperation with 10 different schools (and districts). Two were BIA 
schools (a third BIA school, at Turtle Mountain, declined to participate). The four project 
sites were, at least conceptually, located at four comers of the state, and the director spent 
most of her time on the road, or on the phone, trying to answer questions, resolve problems, 
hire and monitor staff who were both qualified in relevant Ldbjects, including vocational 
education, and able to respond sensitively to community mores and the needs and attitudes of 
the students they were serving, as well as the attitudes and priorities of school principals, 
district superintendents, and others whose cooperation was necessary for the project to achieve 
its overall objectives. 

The challenge of implementing the demonstration on the four reservations was 
intensified by the environmental problems of most of the sites: unemployment was very high, 
ranging from a low of 47 percent at Fort Berthold to a high of 79 percent at Standing Rock 
(Fort Yates). Thus it was difficult for project staff to develop work experience opportunities 
for participants. Further, COFFEE'S entrepreneurial component was not entirely feasible. 
Establishment of student-operated businesses was thought to pose a threat to the few existing 
businesses. Additionally, attendance-or continuity in an educational experience-was a 
problem. The population targeted by the projects was highly mobile-families tended to move 
to Minneapolis and other places to avoid the winter conditions, returning in spring. 
Community attitudes did not always stress the importance of regulai- school attendance. 

Successful project implementation in this environment depended primarily on 
individuals' ability to forge effective informal relationships with officials, staff, and others 
whose cooperation was critical to project implementation. Overall, project location at the 
state level, while it brought the advantage of well-established networks throughout the state, 
also posed problems in terms of dealing with the inevitable day-to-day issues that needed to 
be addressed as working relationships with 10 local schools/districts were developing. While 
the director was generally effective in developing and maintaining viable relationships, she 
resigned from the project at the end of the second year, transferring to another position in the 
State Board for Vocational Education, primarily because the job as director of the dropout 
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demonstration was a temporary one, ending with the completion of the grant period in 
summer 1992.) 

Most of the other projects faced lesser, but nonetheless important, challenges 
associated with the project's location and administrative airangements. The GRADS project 
implemented under a joint arrangement between the Oklahoma Child Service Demonstration 
Center (OCSDC) in Gushing and the Central Area Vocational-Technical School (CAVT) in 
Drumright, is a good example. OCSDC is an organization comprising a group of highly 
trained and experienced educational service providers, most of whom had expensive 
experience in special education, though less experience in working with instructors and 
administrators in an area vocational-technical center. The staff included individuals who 
spent considerable time assisting schools and school districts across the country in 
implementing innovative educational services for adolescents with disabilities, as well as 
evaluation methodologists with expertise in implementing and evaluating educational 
interventions. The participating area vocational school served 1 1 local school districts, the 
largest of which was located in a community of about 6,000 persons. CAVT provided adult 
vocational programs in addition to secondary curricula in most occupational areas. Thus, the 
project's administration was in Cushlng, at OCSDC, while instructional and counseling staff 
were located at CAVT in Drumright, about 10 miles away. 

The administrative structure of CAVT was complex, including an overall 
superintendent responsible for both campuses (Drumright and Sapulpa) of the center, an 
assistant superintendent in charge of the Drumright school, an official in charge of adult 
programs, and a principal for secondary programs. OCSDC staff negotiated CAVT's 
participation with the Drumright campus's assistant superintendent, whose support for the 
project was strong. For a variety of logistical and scheduling reasons, however, the secondary 
principal was not involved in these negotiations and did not originally feel much ownership of 
the project. Because the secondary instructors naturally looked to the principal for guidance, 
this lack of ownership was, at least in the first year, a problem. In the early months, many 
instructors were unwilling to send students to the learning center for assistance in basic skills, 
and some were not willing to accept the project's assistance in making instructional materials 
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more user friendly. Further, a number of instructors would not release students to attend the 
bimonthly group counseling sessions, a key component of the project. The location of some 
of the staff in Gushing and some at CAVT (Learning Center teachers, the counselors) 
exacerbated the problem in that some of the CAVT staff viewed project staff as "outsiders." 
To reduce these barriers, the GRADS project director and other staff undertook specific 
efforts to incorporate the secondary principal into the project's decisionmaking process and to 
win over reluctant instructors. 

While these efforts were largely successful, ongoing difficulties occurred that centered 
around the center's discomfort with the project's group counseling component. The project's 
counselor—trained and credentialed in clinical psychology—was particularly sensitive to 
students' personal issues and tended to take a student-focused approach, while center 
administrators were more traditional in their perceptions of problems that arose. That is, 
given that students must qualify to attend the area center, administrators were able to maintain 
strict disciplinary requirements and tended to view problems as disciplinary rather than as 
personal issues that might require relaxing of the rules in some instances. Again, these issues 
point to the delicacy of implementing interventions in well-established organizational 
structures whose "culture" may be somewhat averse to change. 

It is perhaps instructive that several other project sites that either operated in area 
vocational-technical centers or attempted to develop special arrangements for their participants 
to attend such centers also experienced difficulties resulting in part from the projects' 
administrative structures. Clemson's Anne Arundel (Maryland) and Broward (Florida) sites 
operated in area vocational-technical centers, and both experienced some difficulty integrating 
project activities into that environment. At the McFatter Center in Broward County, for 
example, the school's Executive Director assumed administrative responsibility for the project, 
including decisionmaking regaiding selection of instructional materials, which limited the 
ability of project staff to implement the planned design. The school's director selected a 
computer-assisted instructional program for the applied academics component of the project 



'As is typical in area vocational centers, vocational staff were highly experienced in their occupational fields (the 
requirements for instructo." positions included extensive experience in industry or business) but in some instances 
were less experienced in instructional methods and technology. 
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that became the exclusive instructional vehicle for this component; the system was 
insufficiently flexible to permit integration of academic and vocational programs. As with 
other locations, a lack of consensus about handling of discipline problems and student 
counseling needs limited the project's ability to individualize students' educational 
experiences. Finally, some of the center's instructional staff were not entirely sensitive to the 
special needs and problems of at-risk youth, and project staff struggled to develop strategies 
for orienting staff to these needs in order to improve project participants' success in the 
center. 

Similarly, the Oconee project was not entirely successful in obtaining vocational 
education services for participants. As an alternative school, the project operated in a 
separate location, and students were somewhat isolated from the district's regular high schools 
and the area vocational-technical center. This circumstance resulted in some level of 
stigmatization among the "regular" high school students; additionally, instructional staff at the 
area center were reluctant to make special accommodations for the highly at-risk population 
that the project served. In effect, enrollment at the area center turned out not to be an option 
for project participants. Their vocational component was limited to a school-based business 
and to some community service activities. As noted earlier, one of the North Dakota projects 
experienced a similar problem, although in that instance the issue was not the area center's 
unwillingness to receive project participants but rather the school official's unwillingness to 
send them. 

Many of these project "integration" difficulties were predictable and in fact most were 
eventually worked out over the course of the demonstration. Some, however, were continuing 
challenges that required persistence on the part of project staff. For example, while Sequoia 
Union had a long history of effective implementation of the Academy school-within-a-school 
model, project staff found that each school was unique and that problems attributable to a 
school's administrative structure could arise even after the early stages of implementation. 
While both Woodside and Carlmont enjoyed the full support of the principal (owing in part to 
the very high credibility of the grant's project director throughout the district), other 
administrators were not always supportive. In the Woodside project's third year, an assistant 
principal responsible for the school's schedule resisted arranging the class schedule to enable 
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the senior Academy students to take their senior-level Academy courses. At Carlmont, a new 
assistant principal joined the staff in the project's third year. His views on student discipline 
differed substantially from those of the Academy's lead teacher and staff. Thus, Academy 
students who had experienced a certain set of expectations for their first two years faced new 
requirements, with which many had difficulty. Some felt "betrayed" by the school and by the 
project as well. Thus, while in general the school-within-a-school organization of the 
Academy model works well, probably because of the amount of effort typically directed 
toward developing positive working relationships with administrators and staff of the larger 
school, this model as well as others can face administrative and organizational resistance that 
can affect the experiences and progress of participants. 

Staffing 

As these examples suggest, most of the projects faced challenges in "selling" 
their projects to instructional and other staff who were not employed by the project but whose 
cooperation was critical to project success. Two other types of issues regarding staff were 
important during the projects' early phases, and one continued to be a challenge for most 
projects over their three-year period of funding. These problems included identification and 
hiring of appropriate staff and staff turnover. 

The Oconee County project director, who had long years of experience working with 
highly at-risk adolescents, placed special emphasis on identifying and hiring individuals who 
understood and sympathized with the personal and educational problems of the students his 
project intended to serve. Because the project was located in a rural area, this process was 
time consuming. It was difficult to find the right people-those with appropriate credentials 
who would also be enthusiastic about working in an alternative school with youth whose 
problems were severe. (Many youth were "remanded" to the school and required to remain 
there for at least one semester.) For example, one instructor he hired was a fonner dropout 
who had a good understanding of the problems that can motivate youth to leave school. 
Another, a new teacher, entered teaching after rearing a large family and understood the 
problems that adolescents face in balancing personal and educational issues. The delayed 
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initiation of this project resulted in part from the time required to locate these individuals, 
provide orientation to the project's objectives, select participants, and begin serving students. 

A similar problem faced the North Dakota sites. Because of the extreme dispersion of 
the sites across the state, in combination with very low population density, the director had 
difficulty identifying individuals with the experience and credentials called for by the 
COFFEE model. Although she was able to recruit persons with many of the needed skills, in 
particular with sympathy and understanding regarding the particular problems of the target 
youth, formal credentials as required by the state were often lacking. As a consequence, she 
had to make adjustments in the project model, including abandoning the standard COFFEE 
curriculum and instructional schedule and replacing them with a pull-out model that involved 
remedial tutoring in academics rather than courses for credit. 

The two Sequoia Academies in California encountered a different problem. While the 
lead and other teachers for each Academy were identified and assigned prior to the start of 
the school year, over the course of the first year's operations it became apparent that 
excellence in teaching was not always enough. That is, each Academy had one or two 
teachers who experienced difficulty in working with at-risk students within the Academy 
structure. This issue was particularly difficult at Woodside, whose initial student cohort 
contained a large number of students with low motivation and difficulty in meeting teachers' 
standards. One teacher intern had problems in managing these students, and one of the 
experienced teachers did not acclimate well to the Academy's philosophy. At Carlmont, a 
highly experienced teacher turned out to be unsympathetic to the special problems of at-risk 
youth, setting unrealistically high behavioral expectations and generally refusing to negotiate 
conflicts. At year's end, the project director replaced these teachers with others thought to 
have more potential to work effectively within the Academy model. 

In general, however, the projects were able to identify and employ staff whose 
sensitivity to their participants' needs and ability to devise creative approaches for turning 
disaffected students around were quite remarkable. The North Dakota project director 
routinely traveled several thousand miles a month to develop the rapport with local 
administrators and teachers that she knew were key to the overall success of the project, 
arranging equipment and facilities for project staff, and generally keeping track of the 
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project's widely dispersed sites. The Academies' project director, a long-time district official, 
knew everyone in the district, and used her networks, both in the district and in the private 
sector, to get what the project sites needed in order to get off the ground. Perhaps most 
important. Sequoia was able to find several teachers for each of the sites who clearly loved 
their jobs and were making a difference with their students. Similarly, the staff and 
particularly the counselors at Gushing were able to translate their considerable experience in 
working with special needs students into a set of activities that reflected thoughtful 
understanding of what works with at-risk adolescents. Finally, the Oconee project director's 
sensitivity and understanding of what highly at-risk youth need contributed substantially to 
creation of an environment that he believed would work for students for whom the alternative 
school was a last resort. 

Predictably, over the three years of the demonstration, turnover became an issue in a 
number of sites. Most sites lost one or more teachers, counselors, or support staff, generally 
owing to changes in individuals' personal circumstances (e.g., family relocation). Although 
such losses often caused logistical problems, most projects were able to replace these staff 
without inordinate difficulty. More complicated, however, was turnover among project 
directors. As noted earlier, the North Dakota project director resigned to assume a permanent 
position (the state's gender equity coordinator) at the end of the second year. While the 
replacement was highly qualified and experienced, by then, given the three-year duration of 
the CDP grant, the position was essentially temporary. An official in the state office for 
vocational education was temporarily assigned to complete the grant, and faced the difficult 
task of learning a very complicated and widely dispersed network of people, activities, and 
challenges in a very short period. 

Similarly, the director of the Oconee alternative school resigned toward the end of the 
project's second year; the district made the decision not to replace him but rather to assign 
administrative responsibilities to the district official in charge of dropout prevention. This 
official, having been closely involved in the original grant development, was knowledgeable 
about the project but was unable to assume the level of day-to-day commitment that had 
characterized the prior director. One of the teachers at the school assumed responsibility for 
daily administrative activities. This change in leadership resulted in a redirection in the 
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school's emphasis to some extent. As a special educator with extensive experience working 
with emotionally disturbed adolescents, the initial director had implemented a highly 
individualized, counseling-oriented approach to working with youth, their families, and others 
involved in their problems. While individualization continued to characterize the school's 
philosophy, staff availability to work closely with a large number of students was somewhat 
reduced. 

One of the recommendations from the Clemson project's self-evaluation, which is 

generalizable to all of the projects funded under the CDP demonstration, was the importance 

of recruiting and retaining staff who possessed the skills and motivation to work with at-risk 

adolescents. As noted in its final report, the project recommended that schools considering 

implementing a dropout prevention project should: 

Choose staff from trained, committed, caring volunteers; provide them material, 
professional, and psychological support; and give them the flexibility to make 
important decisions at the local level (Smink, 1992). 

One of the greatest challenges faced by all of the projects was recruitment of individuals with 
the optimal mix of skills, experience, and understanding to address the special needs of the at- 
risk youth who constituted the target population for the demonstration projects. An associated 
challenge was for project staff to respond productively to the views-and sometimes 
prejudices-of their colleagues in the broader environment, including teachers, staff, and 
administrators, as they attempted to implement the project. These findings point up the fact 
that careful project planning, including identification of appropriate staff, is one of the key 
determinants of successful dropout prevention projects. 



Project Participants and Services 



Proiect Participants 

All of the projects participating in the in-depth evaluation targeted services to high 
school-aged youth, with most organized as prevention projects. One project-the Turtle 
Mountain site-originally targeted in-school students but subsequently responded to local 
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needs in extending services to dropouts interested either in preparing for the GED or working 
on credits they needed for graduation in the project's alternative school environment. All 10 
projects used some combination of the "standard" factors associated with risk in identifying 
potential participants: low achievement (grades and credits toward graduation), poor 
attendance, disciplinary or legal problems, behind in school relative to same-age peers, high 
mobility, and poverty. 

Table 3-6 contains the distribution of project participants by several characteristics, 
including gender, race/ethnicity, and age appropriateness for grade. In terms of gender, 
projects located in area vocational-technical centers (Gushing, Detroit, Broward, Anne 
Arundel) enrolled a substantial majority of males (between 70 and 85 percent, on average). 
Overall, only one of the projects (Woodside) enrolled more females than males, a 
phenomenon that reflects the greater likelihood of males than females to be at risk for 
dropping out of school. Ethnic distribution reflected that of the projects' school districts, with 
Detroit enrolling 87 percent black students and Gushing and Oconee 80 and 89 percent white 
youth, respectively. Portland's participants reflected the population of the school in which it 
was located, in that about two-thirds of participants were black. The North Dakota projects, 
located on or near Indian reservations, enrolled nearly all native Americans. The Academy 
projects reflected the changing characteristics of their localities, with enrollment in each fairly 
evenly divided across black, white, and Hispanic persons. 

In terms of age appropriateness, projects that implemented academic requirements as a 
condition of eligibility (or were located in schools with such requirements) more frequently 
enrolled youth whose age was generally consistent with their grade. These included 
Woodside and Carlmont, which imposed academic and behavioral requirements. Additionally, 
the area vocational-technical school in which GRADS operated required entrance require- 
ments. Sixty percent or more of participants in these projects were age appropriate for grade. 
Conversely, a substantial number of participants in most other projects were older. At least 
70 percent of the North Dakota projects' participants were older, as were nearly 80 percent of 
Detroit students and nearly 90 percent of Browaid's project participants. While both of these 
districts had grade point requirements for admission to the area centers, the district agreed to 
relax those requirements for the dropout prevention project participants. 
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Table 3-6 



Characteristics of Project Participants 
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Generally, projects identified eligible youth through referrals from school counselors, 
teachers, or administrators. Additionally, all projects engaged in some form of recruitment. 
Portland's BRIDGE project, for example, served students in ninth and tenth grades, with a 
primary emphasis on ensuring a successful transition from middle to high school for students 
identified as at risk. To be eligible, students had to be between one and four grade levels 
behind in reading or math, but no lower than fourth grade in reading. There were no 
minimum GPA or attendance requirements for entry. Parents were required to attend pre- 
enrollment meetings and to sign a participation agreement. 

In addition to the standard criteria, several of the projects, based on the nature of the 
service design they selected, implemented additional or other criteria for selecting students. 
The two Business Technology Academies developed a screening and application process. 
Although the timing of the grant did not permit full implementation of this process for the 
first-year cohorts, the second-year groups, selected for the 1990-91 school year, were 
screened. To be considered for selection, students had to make an application (complete an 
application form and obtain parent agreement), obtain recommendations from their ninth- 
grade teachers, and participate in an interview with Academy teachers. Because of the 
relative rigor of the Academy curriculum, students were required to have at least a sixth grade 
reading achievement level. They had to display motivation to be in the Academy, and to be 
without serious behavior problems. 

Other projects implemented designs that required accommodation of existing 
enrollment requirements. For example, because it was located in the area vocational-technical 
center that served 1 1 school districts, Cushing's GRADS participants had to meet the entry 
requirements for CAVT. These included being on track to graduate in terms of total and 
distributional credits in order to pursue a vocational program (vocational courses are always 
electives), which required spending one-half of each school day at the Center.* They were 
also required to have the basic academic skills required for completion of the vocational 



''Graduation requirements varied from district to district, which meant that determination of CAVT eligibility 
among students recruited for participation in GRADS was a complex and time-consuming process. 
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program they chose7 Consequently, in effect GRADS' baseline entry criteria included those 
established for admission to the Center. The project's staff attempted to negotiate some 
adjustment to these criteria but were unsuccessful. Their rationale was that the project, whose 
purpose was to encourage improvement and retention among at-risk learners through 
vocational education, could work for students whose level of risk was greater than it was 
among students who normally qualified for CAVT. As was the case in other localities (e.g., 
Ocoriee), however, officials of some area vocational-technical centers resisted enrollment of 
youth whom they believed would be unlikely to succeed in their programs, might be "trouble 
makers," or were otherwise unacceptable. 

In addition to location in areas with high dropout rates, most grantees intended to 
serve youth generally considered to be seriously at risk, by virtue of minority status, overage 
for grade, deficiency in credits toward graduation, disaffiliation with school, and other 
problems typically associated with dropping out of high school. Table 3-7 provides an 
overview of the status of each project's study participants on key measures of school 
performance, in comparison with the performance of "typical" not-at-risk students attending 
the same schools. As shown, in the year prior to enrollment in the CDP demonstrations, 
project participants consistently had substantially lower grade point averages than did their 
classmates, with these differences ranging from .8 to as much as 1.3 points. Similarly, they 
had earned substantially fewer credits than the school's "average" students, had failed more 
courses, and typically were absent more frequently. For example, participants in the Oconee 
alternative school had averaged nearly four times as many absences, those attending 
Portland's Vocational Mentoring project had failed nearly four courses (versus less than one 
for classmates), and nonat-risk students in Detroit had earned more than twice as many credits 
as project participants. Thus, while the severity of youth's academic problems varied across 
sites (e.g., Woodside's students had a 1.32 CPA, versus .84 at Broward and .81 at Fort 
Totten), all were substantially behind in their progress toward graduation. 



'For example, cosmetolDgy was very popular, but required relatively high levels of mathematics and reading 
achievement; consequently, many ("iRADS applicants did not quality fur this curriculum. 
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School Performance of Project Participants and "Average" Students 
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strategies Geared Toward Student Change 



The 12 project sites participating in the in-depth evaluation varied considerably in the 
service configurations they implemented. These variations reflected a number of differing 
perceptions concerning "what works" with at-risk high school-aged youth, what services were 
required to address the specific needs of each project's target population, and the availability 
of instructional facilities and resources in the projects' localities. Because a key component 
of the CDP demonstration was inclusion of a "vocational component," project activities in this 
area were of particular interest. In this section we discuss projects' services. 

Vocational Education and Services 

At the secondary level, "traditional" vocational education tends to take one of two 
forms-and more often than not to be located in one of two types of settings. Other than 
introductory exploratory courses, consumer and homemaking education, and some "shop" 
classes, most comprehensive high schools offer limited vocational programs within the high 
school. These tend to include business and office occupations, marketing and distribution, 
and, more recently, some computer occupations curricula (e.g., CAD/CAM). (Some 
comprehensive high schools do offer the full range of occupationally specific vocational 
programs as well, but many do not, owing in part to resource constraints that limit purchase 
and maintenance of up-to-date equipment and facilities.) 

In the many localities around the nation that do not provide extensive vocational 
programs within the comprehensive high schools, the full range of occupationally specific 
vocational education programs, which prepare students for entry-level jobs or additional 
training at the postsecondary level, is typically available through area vocational-technical 
centers. Students can elect to pursue a vocational program at an area center, spending either 
one-half of each day or, depending on the center's scheduling configuration, every other week 
or two days each week, at the area center. In some states, students attend the area vocational- 
technical center full time for their last two years. In this latter configuration, centers offer the 
academic courses required for graduation at these levels. Students typically attend area 
centers during grades 10 - 12, or grades 11-12, depending on the requirements of the 
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program in which they enroll. Offerings typically include training in trades and industry, 
heaith occupations, food service, i. )tel/moteI management, computer technology, and m.ost 
other occupations relevant to the labor market needs of local areas. A third configuration is 
the vocational high school, such as the Saul School of Agricultural Sciences in Philadelphia 
or New York's Aviation High School. Such schools provide all academic and vocational 
courses needed for high school graduation and achievement of competencies in the vocational 
program within one setting. Finally, many localities are beginning to implement "tech-prep" 
programs, in which a high school links with a postsecondary vocational institution to provide 
a four-year (two in high school and two at the postsecondary level) sequenced occupational 
program leading to an associate degree. 

Other activities-either in addition to or in place of occupationally specific vocational- 
technical education-also fall under the rubric of vocational education. In general, these 
activities are geared toward development of career awareness and employability skills but not 
toward development of entry-level competency in a specific occupation. They include career 
awareness counseling or courses, employability skills development curricula, vocational 
mentoring or advocacy, and nonoccupationally specific, either paid or unpaid, work 
experience or internships. The overall purpose of career awareness activities is to acquaint 
students with the range of occupational opportunities available to them and to help them 
understand the educational requirements of these opportunities. The key intent of 
employability development activities is to "socialize" youth to the worid of work, ranging 
from how to look for a job, interview, and, importantly, how to leave a job, to how to behave 
in the work setting (how to dress, the importance of being on time, how to interact effectively 
with coworkers and supervisors, and other facets of working that many take for granted but 
that are critical to successful participation in the labor force). 

While each of the demonstration projects, as required by the CDP program, 
implemented some form of vocational education, these activities differed substantially in 
content, approach, and intensity. Further, some of the projects for various reasons were 
unable to implement the vocational components as intended and had to miike adjustments in 
their service designs based on problems encountered during the initial phases of project 
implementation. Tabic 3-8 provides an overview of the types of vocational education the 
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Table 3-8 



VocatkMial Components Planned and Implemented by the Projects 
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projects intended to implement, along with an indication of the extent to which they were 
actually able to implement these components fully. 

Four of the projects (Gushing, Detroit, Broward, and Anne Arundel) were located in 
area centers, but their vocational components differed substantially. Gushing' s GRADS 
participants enrolled in the full range of occupational programs available at GAVT. GRADS 
services intended to facilitate students' success in their vocational curricula. Project staff 
worked closely with the vocational instructors to make instructional materials more student 
friendly. This project component included such activitie i as making tests more appropriate 
for students' reading achievement levels, developing computerized, interactive study guides 
for each of the occupational programs at the Genter, providing resource teachers in the 
learning center who worked with students on the academic skills they needed to learn their 
vocational material, and implementing an automated, graduated-level computerized system for 
teaching academic and critical thinking skills in the context of vocational content. While 
convincing the vocational instructors to make use of these materials took time, by the middle 
of the second year most instructors had become convinced of the usefulness of the project's 
contributions, and student mastery of such essential concepts as Ohm's Law improved 
substantially as materials became more widely used and instructors referred students to the 
learning center. It is interesting in this connection that the two GRADS learning center 
instructors, one of whom was a math teacher and the other a science teacher, divided up 
CAVT's vocational programs and undertook to master the vocational content so that they 
could respond effectively to students' needs for assistance. 

Because both treatment and control group members were enrolled in the vocational 
programs at GAVT, many of these project services (e.g., revised materials, learning center 
instruction) were available to control group members as well. The chief exception was the 
computerized system, which was to be reserved for GRADS participants. In fact, one of the 
learning center instructors, who was particularly enthusiastic about this software, was reluctant 
to deny its use to members of the control group. 

Galled Y.E.S. (Youth Experiencing Success), the project at Anne Arundel was initially 
to be a GOFFEE replication. However, funding constraints meant that the site was unable to 
replicate this model, and the site had to make modifications in order to provide services to 
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their participants within the constraints they faced.^ The key service offered to participants 
in the first year of the project was career-related English, which was computer based. The 
curriculum used employability-related topics (e.g., development of resumes and business 
letters) to teach communication skills. Participants received Engiish credit toward graduation. 
A second activity was placement of vocational specialists in the school's shops and labs. 
These instructors worked individually with project participants during the regular class period 
on components of their lessons with which they had difficulty. By the second year, the 
project also implemented a community-based work experience program, under which students 
received credit toward graduation and wei? paid for jobs in the community. Although the 
intent was that these jobs would be in their occupational field of study, most students 
participating in this component were unable to locate such jobs. The full COFFEE model, 
comprising integrated academic and vocational instruction and entrepreneurial businesses 
along with extensive employability development, counseling, and other services, did not 
become available to project participants. 

Similarly, the Broward site did not replicate the COFFEE model, though that site did 
establish a school-within-a-school model at the area vocational-technical center that housed 
the project. Unlike COFFEE, the project did not attempt a student-operated business. 
Further, primarily because of the academic package selected by the school's director, project 
staff were unable to achieve an integrated academic- vocational curriculum for students. They 
were, however, able to implement smaller classes and counseling and other supports for 
participants. 

STARS, the Detroit project, operated in four of the city's area vocational-technical 
centers, with students attending one-half of each day, remaining in their home hi"h schools 
for their academic classes. After extensive negotiation, the project was able to arrange a 
waiver such that home high schools would not lose FTE credit for participants, an 
arrangement that appeared critical to the project's ability to recruit students for enrollment. 

"As one of the Clemson sites, the project received $60,000 in CDP funding for each 18-month period of grant 
funding. The expectation was that participating school districts would contribute additional resources to support a 
relatively faithful replication of COFFEE. Such resources were not available at this site, which was able to 
contribute space and equipment but was unable to add other resources. Consequently, the site implemented a 
substantially different project from the one that was originally intended. 
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Following a traditional model of supplementary services for vocational students, STARS 
instituted instructional support staff for students having difficulty with their vocational 
coursework. Additionally, the project employed persons to assist limited English-proficient 
students with their lessons. Project staff also coordinated students' activities with their home 
high schools. 

In effect, then, the projects operating in the area vocational-technical centers generally 
implemented supplementary services for participants, the purpose of which was to assist them 
in mastering their vocational programs. GRADS differed from the others in having 
implemented a formal group counseling component and in having transformed materials rather 
than limiting services to instructional support and tutoring. Broward varied from the others in 
that the project took the participants' academics to the vocational-technical center rather than 
having participants continue their academics in a regular high school setting. Anne Arundel 
provided vocationally focused English in the area school and credit-bearing work experience. 
Overall, though, the projects did not change the structure of the educational experience in the 
way that some of the others attempted to do. 

As noted earlier, the North Dakota projects involved services to students attending two 
or three of the regular (i.e., not vocational) schools located on or near the state's four 
reservations. Because only a few of the 10 participating schools had access to formal 
occupational training either in the school or in a nearby vocational-technical center, the 
project intended to bring vocational education to the schools through mobile van services. As 
it turned out, this strategy was infeasible, although the project did invest in Apticom, a 
vocational assessment system that helps individuals to identify potential occupational areas of 
interest and aptitude. 

At Standing Rock (Fort Yates) there was no vocational education available to students 
at two of the schools and a limited amount at the third. Thus, the main vocational component 
of the project at this site was adjusted to emphasize career awareness and employability skills 
development, with project instructors working to identify work experience positions for 
project participants. The high unemployment rate and absence of jobs at Standing Rock 
limited project staffs ability to locate work experience positions for project participants. 
Interestingly, the BIA school at Standing Rock contained a number of industrial arts labs, but 
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these were not made available to project participants. In fact, cooperation from the BIA 
administrators was low from the start, and before the end of the three-year period, the school 
withdrew from the project altogether. 

Vocational programs offered by an area center were initially available to project 
participants at Fort Totten, although by the middle of the second year, the superintendent of 
one of the two participating schools withdrew students from the center. Subsequently, these 
students had access to some vocational programs at a nearby postsecondary institution. At 
the Turtle Mountain site, the project's vocational instructor provided vocational instruction in 
Trades and Industry to participants during the first few months of the project. Subsequently, 
he was called to serve in the Gulf War; the project was unable to find a replacement. 
Consequently vocational components were limited to employability-oriented materials and 
instruction made available by the project's academic instructor. At Fort Berthold, vocational 
programs were available to students attending one of the three participating schools. 

As this review suggests, the projects experienced substantial difficulty in their attempts 
to make formal vocational education available to participants. Each site did provide career 
awareness and employability development instruction to participants. However, it became 
apparent that the COFFEE model of student-operated businesses was not feasible at any of 
the project's locations, owing to prevailing labor market conditions. In addition, state-level 
credentialing requirements precluded implementation of the COFFEE model of integrated 
academic and vocational education. 

During the first year, Portland enrolled project students in the BRIDGE progiam, a 
daily one-hour class in prevocational skills. Students participated in BRIDGE during their 
freshman and sophomore years. In addition to following the standard BRIDGE cuiriculum 
(study skills, personal and social development, communications, etc.), students learned 
keyboarding. They spent one period a week working on homework from their other classes, 
and received tutorial assistance as needed. 

The main vocational componeiu funded by CDP that we included in the evaluation 
was Vocational Mentoring, one of several vocational options available to students at Grant 
High School that were oriented to students' particular needs and capabilities. The options in 
the continuum included (1) "mainstream" vocational-technical education for students with the 
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academic- and other skills required to succeed in formal vocational programs; (2) 
"partnership," for students with at least a 2.0 grade point average and a good attendance 
record, which included partnership English and math classes, mainstream vocational programs 
as appropriate, a daily advisory group, and part-time work during the year and a full-time job 
in the summer, as well as other services; and (3) "Vocational Mentoring," for students 
experiencing difficalty in the regular school environment who would benefit from highly 
individualized services and adult mentors. This program operated in cooperation with and 
was located at an area hospital, where students took some academics, were provided support 
services, and worked in a mentoring situation with a hospital allied-health employee. 
Follovying the EBCE model described eariier, this component exposed participants to a variety 
of careers as well as assisting them develop life and employability skills. While students 
received credii fcr successful participation, they were not, strictly speaking, learning an 
occupation. 

Oconee, the third Clemson site, was able to obtain substnntial resources in addition to 
the CDP funding through a grant from the South Carolina state department of education. 
Consequently, the site succeeded in implementing a relatively faithful replication of Project 
COFFEE, operating as an alternative school with an abbreviated school day and other features 
of the model. The school's principal started a business that involved construction, marketing, 
and sale of picnic furniture. This endeavor provided an opportunity for participants to learn 
some vocational skills and to develop employability skills. The project intended to enroll 
students in the area vocational-technical center for more formal vocational training, but met 
resistance on the part, of the center's administrators, who were concerned that the alternative 
school's highly at-risk students would not be able to succeed in the center's environment. 

As noted eariier, the Academies model includes integrated academic and vocational 
training. The Academies implemented at Carimont and Woodside high schools developed a 
business technology curriculum, which included a sequenced program of business courses that 
emphasized computer applications. While the computer component of the curriculum was 
initially somewhat delayed owing to equipment delivery and sei-up logistics, this component 
of the projects was in place by second semester of the first year. In addition, the focus of 
field trips and other activities for students was career awareness and employability 

3-.38 



development, although neither Academy was able to implement a full schedule of such 
activities. Other aspects of vocational services included mentoring and summer jobs for 
participants. These components, as well as other activities in support of the Academy 
projects, were developed in close coordination with the Academies' business partners. 

The latter aspects of the projects' vocational component were not entirely successful. 
Particularly at Carlmont, Academy teachers believed that the "vocational part" did not work 
well. Interestingly, they referred to the mentoring and internship activities rather than the 
business technology curriculum, which they did not exactly view as "vocational education," a 
notion that reflects ongoing stereotypes among academic teachers regarding vocational 
education. In any case, the person responsible for developing internship/work experience 
slots for students had difficulty identifying positions related to students' business technology 
program (primarily because of prevailing economic problems). Further, most students already 
had part-time jobs, often at houdy wages higher than those available through the internship 
positions, and were unwilling to change jobs. 

At both schools, the business technology program was well integrated with students' 
academic work. Students' enthusiasm for the computer lab was reflected in their behavior- 
many spent their lunch periods in the labs working on business technology assignments, or 
their English papers, or other work related to their educational programs. This enthusiasm 
helps to account for the improvements many students made in their school performance over 
the course of the project. 

Academic Services 

All of the projects placed some emphasis on academic preparation, and for most sites 
(though not for the Academies), this emphasis included remediation in basic academic skills 
(reading, math, communications). The vocational and academic instructors assigned to the 
projects in North Dakota provided tutoring to support participants' regular classes, in most 
instances working with the regular academic teachers to identify areas in which participants 
needed assistance. These services were provided on a pull-out basis, with each student 
scheduled to spend at least one hour a week with the academic instructor for tutoring 
assistance. In some siles (e.g.. Fort Yates), the tutors traveled to the students' school or were 
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actually located at the school (Fort Totten). During the first year, the Fort Berthold site tried 
to bus students to a central location. Because of distances (one school was 60 miles away), 
this strategy did not work, and in the second year the project switched to an arrangement 
whereby the two project instructors spent at least one day a week at each of the three 
participating schools. As noted previously, this component differed from the COFFEE model 
the North Dakota project intended to implement, owing to problems with obtaining official 
pilot status for the project. 

The Turtle Mountain project arranged facilities in a "store front" location, with 
students spending their project time at that facility, working individually with the academic 
instructor. Because many of these students were dropouts working toward a GED, this site 
did not face the same difficulties regarding graduation credit, and the project was more nearly 
like the COFFEE model than were the other three sites. One of the most interesting features 
of this site was the instructor's development of a culturally sensitive, integrated academic and 
employability-related curriculum for her students. She tried, eventually unsuccessfully, to 
arrange related work experience opportunities. 

As noted earlier, part of the program for Portland's BRIDGE students involved tutorial 
assistance in academic subjects, and Vocational Mentoring included an applied academics 
component that students took on site at the participating hospital. This curriculum integrated 
life and employability skills with the academic coursework. Anne Arundel implemented a 
credit-bearing career-related English course for participants. 

The two Academies, Broward, and the Oconee County alternative school provided the 
full academic program that students needed to graduate, although in their later high school 
years, Academy students took some of their academics in regular classes. Broward used 
almost entirely a computer-assisted program for academics, though the project did employ 
academic teachers to work with students on an as-needed basis. All four projects 
implemented reduced class size. Additionally, the Academies developed enriched curricula 
that integrated occupational and academic components, with students working on social 
studies and English papers in both the academic and business technology classes. Oconee 
also provided all academic courses needed by participants but encountered some difficulty 
meeting the state's physical education requirements because of lack of facilities. 
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Counseling and Other Support Services 

Based on the perception that by high school many at-risk students have become 
seriously disaffected with, or even alienated from, education owing to a long experience of 
failure, some of the projects implemented a counseling component. These services ranged 
from career awareness counseling to provision of formal personal counseling in individual or 
group settings. For example, one of the key services at Gushing was formal bimonthly group 
counseling for all participants, supplemented by individual counseling and referral to 
community services as needed. Topics covered ranged from personal and school adjustment 
issues to career plans. While this component was key to the GRADS service design, project 
staff experienced some difficulty in getting it fully implemented. A number of the vocational 
instructors were reluctant to release their students from class to attend the groups, and project 
staff worked to change these attitudes in order to ensure the effectiveness of this component. 

The Oconee County altemative school, which enrolled highly at-risk students, 
employed a counselor who worked primarily in the area of career guidance. However, this 
person did provide some personal counseling, and the project's director also spent a 
considerable amount of his time providing individual counseling to project participants on 
such topics as school behavior, legal issues, or personal problems students were dealing with 
at home and with their peers. The intent of these services was to create an environment in 
which students could begin to identify and affiliate with the school and thus improve their 
attendance and achievement levels. Departure of the principal toward the end of the second 
year meant that this level of personal assistance was no longer possible at the site, a change 
that probably constrained some students' further progress. 

While the Sequoia Academies did not include a formal counseling component, the 
Academy at Woodside employed an individual who in reality served this function on an 
informal basis. She monitored students' progress through attending classes and bringing 
students in for extra assistance as required. A resident of the community in which most of 
the project's participants lived, this person also implemented a proactive attendance 
monitoring program, checking daily on students' presence at school and presence in classes. 
She called students in, talked with parents both on the phone and in person, visited homes, 
and generally kept up with each student's progress in the project. In the process, she 
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counseled students on behavior, personal problems, and other issues with which they needed 
assistance. 

All projects provided at least some informal counseling-oriented support to 
participants, typically in the context of tutoring assistance as with Anne Arundel and the 
North Dakota projects. Inevitably, the project staff, in working closely with small numbers of 
students, became involved in their personal as well as academic problems. This aspect may 
have been one of the most useful components of the projects, providing youth the opportunity 
to receive attention and concern from adults. The Portland project, in fact, formalized this 
strategy in the vocational mentoring activities, with mentors assisting students with various 
aspects of their problems. 

In addition to counseling, several of the projects made other supportive services 
available to students. These included child care for teen parents (Gushing), case management 
and referral (Oconee, Gushing, and Portland), transportation (Turtle Mountain, where the 
instructor often drove to participants' homes to take them to school). Further, most of the 
projects provided some form of student incentives. The four North Dakota sites, for example, 
paid students for attending school; at the rate of one dollar per day, participants could earn up 
to about $25 per month for attendance. Other sites provided parties or field trips. Finally, 
the work experience jobs for Academy students were intended as incentives. Only students 
whose school progress was acceptable were permitted to participate in this project component. 



Structural Changes in the School 



As noted earlier, many of the projects undertook some form of alternative 
environment, based in part on the understanding that the youth they targeted for services had 
not fared well in the regular school environment. The Oconee and Turtle Mountain projects 
were housed in separate facilities. The two Academies and Broward were structured as a 
school-within-a-school, and both Fort Berthold and Fort Totten located some staff in 
alternative facilities (a trailer in New Town and a house behind the Four Winds community 
school), 
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Oconee, Turtle Mountain, and the Academies (and to some extent Broward, though in 
that case students took their vocational courses in the "regular" vocational-technical 
environment) represented major restructuring of participants' educational environments, while 
the other projects could be classified as supplemental to participants' regular school programs. 
Implementation of alternative structures and environments has both advantages and 
disadvantages for at-risk students. On the one hand, alternative schools offer the advantage of 
providing an environment in which different approaches can be tried in attempting to reach 
youth; classes are generally smaller, and the formal structure of regular school can be relaxed 
in order to provide an experience that may be more productive for students. (Oconee, for 
example, obtained some latitude from the district in dealing with attendance and disciplinary 
problems, which probably could not occur if the school were located at one of the district's 
"regular" high schools.) This advantage applies to the school-within-a-school model as well, 
in that students have a relatively smaller context with which to identify. 

On the other hand, this type of structure, particularly the alternative school, does 
segregate students, and stigma can be a problem. Students at Oconee, for example, rode 
buses to the regular schools, and then were bused to OASIS, which meant that their former 
classmates were aware that they were going somewhere else, and students reported that they 
experienced discrimination as a result. According to the project's director, however, students 
were so "desperate" that they were willing to endure discrimination and stigma in order to 
attend the school. 

According to several of the projects' directors, the advantages of alternative schools or 
settings outweighed the drawbacks. These youth needed a new educational environment, 
including more attention, positive reinforcement, and nurturing, in combination with clearly 
articulated and enforced behavioral expectations, if they were to make it to graduation. 
Further, in this environment, teachers and other staff could work more closely together to 
monitor students' progress, achieve earlier identification of problems, and address issues 
before they became so serious as to result in participants' dropping out, or being expelled, 
from school. In short, the alternative model changes the entire environment, while projects 
that provide supplementary services may affect only parts of participants' school experiences. 
The effects, and effectiveness, of the projects' service configurations in the broad sense, 
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including what they did for and with students and in what environments, are the subject of 
Chapter 4. Chapter 5 synthesizes the study's findings and discusses their implications. 
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CHAPTER 4: RESULTS 



Introduction 

Examination of the effects of the demonstration projects on participants' academic 
performance, school affiliation, and perceptions in selected areas relevant to school must 
occur in the context of the projects' objectives, activities, and environments. Some of the 
service designs selected for implementation by the grantees were more comprehensive and 
coherent than others, and thus potentially more likely to change the overall quality of 
students' educational experiences and outcomes. This variability in design must be 
considered in the context of two important factors: (1) the nature and extensiveness of the 
problems of projects' target populations, and (2) the likelihood that even intensive and far- 
reaching interventions, such as alternative schools that address youth's personal as well as 
educational difficulties, may have some limitations (e.g., isolation from nonat-risk peers, 
stigma). Even within these constraints, the organization and coherence of projects' service 
designs, along with the extent to which they attempted to meet multiple needs of students, 
appeared to affect th-:; types and extent of outcomes that resulted from participation. 

All of the models selected for demonstration appeared to have good potential to 
improve educational experiences and outcomes of participants. However, problems that 
several projects experienced in getting underway, delays in setting up computer labs or 
obtaining equipment, and changes over the course of the demonstrations in staffing, 
administrative support, and other factors affected the types of achievements some of the 
projects were able to demonstrate. Additionally, in some instances local factors meant that 
substantial changes in intended models were necessary, and the longer term implications of 
these changes for student outcomes must be considered as the overall success or failure of the 
demonstration is considered. 

In this chapter, we provide a summai7 of the projects' outcomes and offer some 
suggested explanations, based on our observations of the projects' activities over their 
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existence (some for three, others for only two, years), as to why these outcomes may look as 
they do. As shown in Table 4-1, outcomes are classified into three categories. "School 



Table 4-1 

Types of Participant Outcomes Included in the Evaluation 
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More credits earned 
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■ Better future 
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Fewer absences 
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Counseling/counselors better 


expectations 
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Fewer dropouts 


■ 


More academic encouragement 


■ Classmates are 










college bound 
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performance" outcomes include measures of student academic and behavioral status, including 
grade point average, number of credits earned, number of courses failed, number of absences, 
and dropout status. Improvement in these outcomes suggests that the project's intervention 
had effects on students' ability to perform acceptably~and stay-in school. The second 
category, "school affiliation, " includes measures of students' perceptions that their school 
experiences were positive-they made positive judgments about school staff (teachers and 
counselors) and their interactions with these staff; they believed that the school environment 
and discipline were productive for them; they believed that what they were getting from the 
school (preparation for jobs, assistance in planning and participating in their educational 
courses, and encouragement in their class work) had improved since the inception of their 
participation in the project. Finally, "student perceptions" include several measures of the 
implications of the educational experience for students' perceptions regarding their 
environment. 

The results of all of the analyses discussed in this chapter are presented in Appendix B 
as a series of tables containing findings for each of the 12 sites that participated in the 
evaluation. These summary tables include the following: 

• Selected demographic characteristics of study participants 
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• Analysis of school performance outcomes, for all cohorts and all years 

• Gap reduction analysis of school performance outcomes, by cohort by year 

• Analysis of project dropout rates, by cohort by year 

• Analysis of school affiliation and student perception outcomes, by cohort by 
year 

• Gap reduction analysis or school affiliation and student perception outcomes, 
by cohort by year. 

The tables present results for the groups of treatment and comparison/control students 
referred to as Cohort 1, Cohort 2, and a "combined" cohort. Cohort 1 is the group of students 
selected for participation in the study beginning in school year 1989-90 in the projects that 
were able to get underway that fall after receiving the CDP grant in July 1989. The study 
followed Cohort 1 students across three school years: 1989-90, 1990-91, and 1991-92. 
Cohort 2 comprises either a second group of students selected from the sites that were able to 
begin in fall 1989 (and followed for the 1990-91 and 1991-92 school years) or the only group 
of study participants for projects that began serving students in spring or fall 1990. Finally, 
to increase the overall sizes available for analysis, we constructed two "combination Cohorts" 
These are the Cohort 1 and Cohort 2 year one group (i.e., the first year of each group's 
participation in the "treatment" or comparison condition) and the Cohort 1 and Cohort 2 year 
two group (their second year of participation). 

All results were adjusted to remove any effects resulting from differences between 
treatment and comparison/control groups in gender, race, or age. Results are based on raw 
scores rather than containing imputed data. The findings presented in the chapter represent 
statistically significant differences at the .05 level. Any results not mentioned can be assumed 
not to have been significant and reliable. Finally, results obtained in 1991-1992 are not 
independent from results obtained in 1990-1991 for the same cohort, because dropouts and 
gains were measured against the same baseline (school year 1988-1989 for Cohort 1 and 
1989-1990 for Cohort 2) in both cases. Further, the results for the two time periods may 
differ because some students available for comparisons during earlier years were not included 
in later analyses, having transferred, graduated, or dropped out. 



Table 4-2 contains a summary of the evaluation's findings regarding participant 
outcomes (as noted previously, details of these findings appear in Appendix B). This 
summary lists significant positive outcomes for each site, by cohort and by year. The body of 
the chapter contains details of our findings for each of the 12 sites participating in the 
evaluation. 

We have elected to organize the discussions by site to assist the reader in comparing 
the outcome data presented here with the process data presented in Chapter 3. In considering 
the discussion of these outcomes, the reader is cautioned that changes in student performance, 
affiliation, and perceptions may require a considerable period to occur. Particularly for older 
students, interventions must affect what may be years of unproductive and sometimes painful 
educational experiences, and a predictable progression is often that changes in student 
attitudes about and affiliation with school may substantially precede measurable changes in 
school performance. 

Our discussion of results for each project includes a brief summary of the project's 
service design. Additionally, based on our analysis of the nature, intensity, and 
implementation levels of the service designs of the projects, we articulate what appear to us 
to be "plausible" outcomes~that is, the types of effects that the projects might be expected to 
have on the majority of students receiving the interventions. Specification of these outcomes 
for each of the projects participating in the evaluation was based on the totality of that 
project, including consideration of the configuration of services, experience and stability of 
staff, types of students targeted for participation, coherence and intensity of services, and 
judgments, based on interviews with staff and students as well as observation of project 
activities, concerning the types of outcomes that a project could reasonably be expected to 
achieve within the framework of the interventions provided. 



4-4 



ERIC 



Si 




Woodtidt Acadtmy 
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■ Higher GPA 
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First-year results 
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■ More credits 

■ Fewer courses failed 

■ School thought safer 
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Second-year results 
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Hrst-year results 
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Second-year results 
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Woodside Business Technology Academy 
Woodside Hieh School. Woodside. California 

Overview 

Following the Academy model, the Business Technology Academy at Woodside High 
School was organized as a "school-within-a-school." Students entered the program at the 
beginning of tenth grade; capacity was about 50 students per grade. Participants stayed 
together for most of the school day, taking core classes (English, mathematics, social studies, 
and business technology) together, and with the same set of Academy teachers, over the 
three-year period. Classes were small, and teachers received an extra preparation period to 
coordinate curriculum across academic and vocational subjects, plan extracurricular activities 
(e.g., the mentoring program, field trips), and consult about students' progress and needs. 
The program targeted students who were moderately at risk, with eligibility based on a 
specified reading achievement level, motivation to participate, and absence of serious 
behavioral or emotional problems. In addition to this "standard" Academy model, Woodside 
employed an instructional support specialist who monitored participants' attendance, 
completion of homework, and classroom behavior. 

Across the two study cohorts. Academy students differed somewhat in terms of 
selected demographic characteristics (gender, race/ethnicity, age). A majority of Cohort 1 
study participants were male (56 versus 44 percent), while for the second cohort, substantially 
more were female (60 versus 40 percent). For both cohorts, more students were Hispanic (37 
and 34 percent, respectively) than other ethnicities, with 32 and 29 percent black, and 29 
percent white in both cohorts. Finally, nearly two-thirds of Academy students in both study 
cohorts were age appropriate (i.e., were rhe same age as most students in their grade). Table 
B-1.1 in Appendix B provides comparable information for the site's comparison group 
members. 



Plausible Outcomes 

Participation in (and completion of) the business technology program as implemented 
at Woodside could be expected to eventuate in the following outcomes: 



• Improvement in school performance, including attendance, grades, and credits 

• Reduction in dropping out among participants 

• Improvement in postschool outcomes, including employment in entry-level jobs 
with career potential and enrollment in postsecondary school. 

Results 

The Woodside Academy began operations in fall 1989, with assignment of students 
selected for the Academy as tenth graders to the "treatment" group and selection of a matched 
comparison group of students who did not enter the program. The study collected 
information about these students over a three-year period, ending when they would have been 
expected to graduate from high school in spring 1992. A second cohort, who entered as tenth 
graders in fall 1990, was followed for a two-year period, along with a similarly selected 
comparison group. Analyses reported here are for each of the three years of Cohort 1, the 
two years of Cohort 2, and the first and second years' experiences for the "combined Cohort," 
which reflects the first and second years' participation of both cohorts. 

Baseline to 1989-1990 } Business Technology Academy Cohort 1 students at 
Woodside High School achieved improvement in school performance during 1989-90, 
measured against their performance in the prior year (school year 1988-89). These 
improvements included higher grade point averages (GPA) and an increased number of credits 
earned. The GPA improvement was almost two-thirds of a grade point. The treatment group 
moved up from a pretreatment grade point average of 1.29 (slightly below a D-h) to an 
unadjusted posttreatment grade point average of 1 .94 (slightly below a C)— a nontrivial gain. 

Additionally, participants' perceptions of their school experiences resulted in some 
changes in school affiliation: they perceived the school environment as safer, that their 
teachers and instruction were better, and that their classmates had more diverse friendships. 
Specifically, both covariance analyses— but not the gap-reduction analysis— found a significant 



'Analyses of 1989-1990 data included analysis of covariance (ANCOVA), reliability-adjusted analysis of 
covariance, and gap reduction analysis. Analysis of subsequent years' data included gain score analyses in addition 
to those techniques used in analyses of the first year's data. 

4-8 



ERIC 



92 



improvement in feelings of school safety and attitudes toward teachers and teaching. The 
same pattern of results was observed with respect to perceptions that students made friends 
with students of other racial and ethnic groups. 

The Academy's school- within-a-school model, which emphasized smaller classes, 
more coordination among teachers in working with individual students, increased attention to 
students' day-to-day progress among teachers and particularly from the project's instructional 
support specialist, and provision of tutorial assistance, may help to account for these findings. 
Additionally, block scheduling of classes, in which students remain with their classmates and 
a core set of teachers for most of their school day, along with incentives and other activities 
intended to foster a more efficacious environment, may have contributed to students' 
improving affiliation with school. 

Baseline to 1990-1991 . Cohort 1, which achieved positive academic and other 
outcomes in their first year, did not demonstrate significant outcomes from baseline to the end 
of year 2. However, Cohort 2, which entered the Academy in fall 1990, demonstrated 
significant effects in grade point average (which increased from 2.0 to 2.4, versus the 
comparison group's decline from 1.8 to 1.7) and total credits earned (from 24.3 to 28.2, 
versus the comparison group's decline from 24.3 to 22.7) during their first year of enrollment 
in the program. Additionally, comparison group students failed more courses (up from 1 .0 to 
2.0), while Academy students remained constant at 1.0. Finally, gap reduction analyses, 
which compared treatment students' performance with a sample of "average, not-at-risk" 
students, showed significant positive outcomes for Academy students on grade point average 
and number of courses failed (which declined), for both cohorts. 

Expectably, the Year 1 experience of the two cohorts combined showed positive 
outcomes for grade point average, number of credits earned, and reduction in number of 
courses failed. On other measures, analyses of the combined cohort showed a significant 
increase in participants' feeling that school was safe. Cohort 2 students increasingly believed 
that their classmates were college bound (one of the measures of students' changing 
perceptions). However, gap reduction analyses suggested that Cohort 1 students experienced 
reduced expectations for the future in their second year in the program. 
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Baseline to 1991-1992 . Positive outcomes continued to erode for Cohort 1 students 
during their third year in the Academy, particularly in the area of academic performance. For 
example, their grade point averages did not increase, as opposed to the averages of 
comparison students. Further, comparison group members improved from failing an average 
of 2.1 courses to an average of 0.3, while Cohort 1 Academy students remained stable at 1.8 
course failures, on average. 

Similai-ly, for Cohort 2, treatment students were absent somewhat more frequently (2.0 
to 4.5 days, versus 2.3 to 4.3 among comparison students). On the other hand, treatment 
students were more likely to perceive that they were in classes with students preparing for 
college (from 2.2 to 2.7 on a four-point scale) versus other students (2.7 to 2.6). 
Additionally, gap reduction analyses were positive for the combined cohort in their second 
program year for increased grade point average and reduced number of courses failed. 

Droppins out . Analysis of data from this site revealed no significant differences in 
dropping out across years or across single or combined cohorts (Table B-1.2). As shown in 
Table B-1.4 (Appendix B), very few students in either cohort's treatment or comparison 
groups dropped out of school. Rates were highest in Cohort I's third year, with four 
treatment students dropping out by their senior year (school year 1991-91), for 10 percent 
overall, and five comparison group students (1 1 percent) having dropped out by that year. 

Summary 

These results suggest that the Woodside Business Technology Academy experienced 
some success in terms of improving participants' academic performance, although these 
positive outcomes tended to decline over time (three years of data collection for Cohort 1 and 
two years for Cohort 2). The project experienced more success with first-yeai- than with 
second-year students, although in comparison with the gap reduction group, positive outcomes 
for grade point averages and reduced number of courses failed continued across all years. 
This relative decline in significantly positive outcomes may reflect students' having become 
"accustomed" to the special features of the project, thus reducing what was likely a 
Hawthorne effect during their first year. Additionally, school affiliation outcomes tended to 
be stronger during students' earlier yetirs of participation than later. The project did not 
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appear to exert a significant influence on students' likelihood to drop out of school, although 
for both treatment and comparison groups, the numbers of students who left school before the 
end of the study were quite small. This finding is explained in part by the fact that students 
entered the Academy in the tenth grade, a year later than the critical "dropping out" year that 
typically occurs before, during, or just after the ninth grade when many students become old 
enough to drop out legally. 

Observations and discussions with faculty, administrators, and students at the 
Academy over the three-year period do suggest, at least anecdotally, that participation in the 
project had a salutary effect on many students as well as most of the staff who served as 
Academy teachers and support persons. Students appeared generally more engaged in their 
classes and independent work, and teachers who remained with the project demonstrated a 
high level of commitment to the model. The implementation problems, along with somewhat 
less than optimal support from some administrators (see Chapter 3), may have reduced the 
project's positive effects. Similarly, the evaluation model (a matched comparison group 
design) implemented was less than optimal and may help to account for the modest effects 
that we found. 

Carlmont Business Technology Academy 
Carlmont High School. Carlmont. California 

Overview 

Like the project at Woodside, the Business Technology Academy at Carlmont featured 
small classes, block scheduling of students, an admissions procedure intended to maximize 
students' likelihood of success, and extracurricular activities coordinated with participants' 
vocational courses. The Academy model emphasized careful selection of teachers, assignment 
of a lead teacher to manage the project, and coordination of curriculum across academic and 
vocational components. The two Academies shared a career development specialist, whose 
responsibilities included recruitment of business partners and mentors (the mentoring program 
began in eleventh grade), development of work experience and internship slots for students 
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following eleventh grade and in the second semester of theu- senior year, and monitoring of 
student progress on their jobs. The Carhnont Academy did not have a staff person analogous 
to Woodside's instructional support specialist. 

The Carlmont project's two study cohorts differed somewhat in demographic 
characteristics. A substantial majority of Cohort 1 Academy students were male (63 percent), 
while the majority of Cohort 2 students were female (52 percent). For both treatment cohoivS, 
more students were white than other race/ethnicities (39 percent for Cohort 1 and 38 percent 
for Cohort 2), with 29 percent black in each cohort and around one-fourth Hispanic. Finally, 
nearly three-fourths (71 percent) of Cohort 1 students were age appropriate, compared with 
only 52 percent of Cohort 2 students. Table B-2.1 in Appendix B provides comparable 
information for the site's comparison group members. 

Plausible Outcomes 

Outcomes that could be expected to be associated with the Carlmont project included 
the following: 

• Improvement in grades and credits earned 

• Improvement in school affiliation 

• Increase in postschool aspirations 

• Increase in postsecondary enrollment 

• Reduction in dropping out. 

Results 

Carlmont' s Business Technology Academy began operations in fall 1989, with 
assignment of students selected to participate as tenth graders to the study's "treatment" group 
and .selection of a matched comparison group of Carlmont students who did not enter the 
project. The study collected information about these students over a three-year period, ending 
in spring 1992 (when most remaining participants graduated). A second cohort, entering as 
tenth graders in fall 1990, was followed for a two-year period. Analyses reported here are for 
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each of the three years of Cohort 1, the two years of Cohort 2, and the first and second years' 
experiences for the combined cohort. 

Baseline to 1989-90 . Carlmont's Academy students also improved their performance 
over the course of the school year (higher GPA and reduction in the number of courses 
failed). However, a significant improvement in grade point average was found only in the 
gap-reduction analysis. A significant reduction in the number of courses failed was found in 
the reliability-adjusted covariance analysis and the gap-reduction analysis, but not in the 
regular covariance analysis. These differences were caused not by a reduction in the number 
of courses failed by members of the treatment group, but by an increase in the number of 
courses failed by members of the comparison and gap reduction groups. 

Additionally, participants' school affiliation increased, including their perceptions of 
their teachers and instruction, their perceptions about the safety of the school, and their 
perception that they were receiving g'-od job preparation and more encouragement regarding 
their academic course work. Further, they perceived that many of their classmates were 
college bound. According to project staff, their increasing awareness of college was 
explained in part by these students' emerging perception that college might be an option for 
them, and they became more interested in the postschool plans and aspirations of their 
classmates. 

As with Woodside, Carlmont's intervention stressed a smaller, more cohesive 
environment for participants, with substantially more attention, and also higher expectations, 
from adults than many of these students had experienced in the past. Several of the Academy 
teachers at Carlmont spent a great deal of their own time, in addition to their school time, 
with these students, taking them to plays and other activities, talking with them about 
problems and interests, and treating them with considerable respect and understanding. These 
aspects of the project at Carlmont may help to explain the first-year findings of the 
evaluation. Additionally, as noted in Chapter 3, students at both Academies experienced a 
full year of the project; while some of the components were not fully implemented at the 
beginning of the year, nevertheless, students had a relatively longer period in which to begin 
to show effects of participation. 
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Baseline to 1990-1991 . Academy students in Cohort 1 continued to experience 
academic improvement during year two, increasing their grade point averages (from 1.5 to 
1.8), while comparison group students declined (from 1.7 to 1.5). Similarly, treatment 
students failed fewer courses (from 1.6 to 1.3) than did comparison students (who increased 
from 1.1 to 2.1). Compared to gap reduction students, both cohorts, singly and in 
combination, improved in grade point average, and Cohort 2 failed fewer courses during the 
year. 

In addition, Carlmont students had positive outcomes on other measures. For example. 
Cohort 2 students thought their teachers and teaching were better, and Cohort 1 students, 
compared with the gap reduction group, believed they were receiving better job preparation. 

Baseline to 1991-1992 . Cohort 1 treatment students experienced increased grade point 
averages (from 1.5 to 2.2) with the increase exceeding that of comparison students (1.7 to 
2.0), and the combined cohort increased GPA as well. The combined cohort reduced course 
failures (1.4 to 1.3, versus to 1.2 to 1.9 for the comparison group). However, they were 
absent more often. The combined cohort believed that their classmates should not misbehave. 

Dropping out . Analysis of data from this site revealed a significant difference in 
dropping out for Cohort 1 in their third year and for the combined cohort in year two. The 
project was particularly effective with overage students in Cohort 1: seven of 19 overage 
comparison group students dropped out, while none of the 12 overage Academy students 
dropped out. As shown in Table B-2.4, dropout rates for comparison group students 
substantially exceeded those for the treatment group. For Cohort 1, cumulative dropout rates 
(at the end of the third year) were zero for treatment group members and 19 percent for the 
comparison group. For Cohort 2, comparable rates were nine percent and 15 percent at the 
end of the second year; for the combined cohort, rates were six and 16 percent, respectively. 

Summary 

Overall, students enrolled in the Business Technology Academy at Carlmont High 
School improved their academic performance as a result of their participation, specifically in 
terms of improved grade poim average and reduced nimber of courses failed. The project 
was one of the few in the demonstration overall for which dropping out was significantly 
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lower among participants than among comparison/control group members. The fact that these 
improvements were relatively stable over time may reflect the high level cf staff involvement 
in and commitment to participants' academic and personal development. As noted in Chapter 
3, faculty engagement in the project tended to increase over time, with changes in the 
composition of the faculty at the end of the first year having a salutary effect on student- 
teacher relationships, resulting in high levels of faculty commitment among all Academy 
teachers during the demonstration. Administrative support over the project's first two years 
was consistently strong. In addition to evidence of the project's success in terms of academic 
performance, the fact that some students elected to enroll in higher level courses, particularly 
in mathematics, along with their frequent attendance at the computer lab during lunch and 
free periods, suggests the efficacy of the project in engaging participants in their education. 

Guidance in Retraining Adolescents (GRADS) 
Central Area Vocational-Technical School. Cashing. Oklahoma 

Overview 

Guidance in Retraining Adolescent Dropouts (GRADS) was a cooperative effort 
involving the Oklahoma Child Service Demonstration Center as the grantee and th^ Central 
Area Vocational-Technical School (CAVT) as the project site. Originally developed to assist 
students with learning disabilities to improve their academic achievement, GRADS used 
prescriptive teachers to adapt instructional materials for students with special learning needs 
and to train regular vocational instructors in appropriate instructional methods. The project 
supported a resource center at CAVT that included a professional staff and computer-managed 
instructional support, affective interventions whose purpose was to improve students' self- 
concept and school affiliation, and inservice training for vocational instructors. The project 
targeted moderately at-risk youth who were still in school. Participants attended high school 
in one of 1 1 school districts that were feeders to CAVT. Originally intended to target highly 
at-risk youth, the project was required to modify this goal because youth had to be 
academically eligible to attend the area vocational-technical school in order to participate. 
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Key features of the project included twice-monthly group counseling for all participants and 
indivic'ual counseling as needed, adaptation of instructional materials for all vocational 
programs in which participants enrolled, computerization of vocational curricula to enable 
students to work on coursework in the resource center, and inservice training and support for 
vocational instructors to facilitate their success with at-risk youth. 

Across the two study cohorts, GRADS participants were predominantly male (78 and 
64 percent, respectively) and white (78 and 83 percent). Eighteen percent of Cohort 1 
treatment students and 15 percent of those in Cohort 2 were Native American. These 
characteristics reflect the enrollment profile of the area vocational-technical center. Two- 
thirds of Cohort 1 students were age appropriate, while a slight majority (53 percent.) of 
Cohort 2 students were age appropriate. Table B-3.1 in Appendix B provides comparable 
information for the site's control group members. 

Plausible Outcomes 

Participation in GRADS could be expected to result in the following outcomes: 

• Improvement in vocational course grades and vocational skills 

• Reduction in dropping out of school (and consequent increase in graduation 
rate) 

• Development of more realistic postschool expectations 

• Improvement in postschool employment prospects. 

Results 

Gushing, Oklahoma, a random assignment site, began operations in fall 1989, with 
random assignment of students to treatment or control groups and selection of an additional 
group of students for the gap reduction group. A second cohort entered the project (and the 
study) in fall 1990. Gushing was different from some of the other sites in that all students 
participating in the evaluation, including the treatment, control group, and gap-reduction 
students, spent half of each day in a vocational-technical school where they pursued a 
vocational program (e.g., electricity, motorcycle mechanics). The area school by itself may 
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have an impact on student performance independent of the treatment, which was an add-on to 
that component. 

Baseline to 1989-1990 . GRADS participants at CAVT experienced significant 
improvements in school performance in comparison with gap reduction students, though not 
in comparison with control group members. The treatment group gained significantly in 
terms of grade point average and credits earned, and reduced significantly the number of 
courses they failed. Gap reduction analyses reflected a perceived decrease in the safety of the 
school, while both analyses of covariance reflected a reduction in the perception of having 
received a lesser amount of academic encouragement. 

While the instructional support services available to participants were also available to 
gap reduction and control group members, GRADS participants did receive services not 
available to other students, including particularly a strong group counseling component and a 
software system that integrated academic and critical thinking instruction with vocational 
content. However, neither of these components was fully available to participants during 
school year 1989-90. Many vocational instructors resisted sending their students to the 
groups, so a substantial number of participants did not really receive this service until toward 
the end of the school year; and some did not receive it at all. The software system was not 
available until near the end of the first semester, and some confusion about its availability 
meant that its use was not entirely restricted to GRADS students. 

Baseline to 1990-1991 . For both cohorts, gap reduction analyses demonstrated 
continued improvements in academic performance, including higher GPA, higher numbers of 
credits earned, and fewer courses failed. The combined cohort also reduced number of 
courses failed on the gain score and both covaiiance analyses. In terms of school affiliation. 
Cohort 1 showed a perceived improvement in future expectations. 

Baseline to 1991-1992 . Cohort 2 and the combined cohort for year two continued 
academic improvements in GPA, credits earned, and reduction in courses failed, compared to 
the gap reduction group. Analyses for Cohort 1, however, failed to show continued academic 
improvement in the third year. The only school affiliation measure with a positive outcome 
for this year was students' perception of better job preparation for Cohort 2 as well as for the 
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combined cohort's second year, compared with the gap reduction group. Cohort 2 exhibited 
reduced expectations for the future in this year in comparison with the control group. 

Dropping out. Cohort 2, and the combined cohort for year 2, experienced significantly 
lower incidence of dropping out of school, indicating that the GRABS project achieved 
success on this measure after the project became fully implemented. Dropout rates for these 
cohorts, as shown in Table B-3.4, were as follows: four percent for the treatment group 
versus 19 percent for the control group among Cohort 2 students in the second year, and 10 
versus 22 percent for the combined cohort's second year. 

Summary 

GRADS was one of the few demonstration projects that achieved significantly lower 
dropout rates among project participants. In addition, the project experienced posit.Ve 
outcomes in school performance in comparison with gap reduction students, although less so 
in comparison with control group students. The structure of the project, along with attitudes 
of some of the vocational teachers, may have constrained project effects somewhat. While 
substantial effort went into preparation of more user friendly instructional materials, including 
special computerized tutorials, to support students in their vocational programs, these 
materials were made available to all students in all programs open to high school youth at the 
vocational-technical center. Consequently, to the extent that the materials were helpful, all 
students (treatment, control, and gap reduction students) could have benefitted, thus 
potentially masking treatment effects in the area of school performance. The fact that the 
project achieved significant outcomes in retention of students suggests the efficacy of its 
support components, in particular the group counseling sessions at \ hich students were 
encouraged to work on personal and other problems that interfered with their progress in 
school. Finally, the fact that the project's counselor, in addition to running the sessions, 
worked with individual students on myriad problems, from housing to disciplinary issues, 
probably helps to account fcr the project's success in keeping participants in school. 
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OASIS Alternative School 
Oconee County, South Carolina 

Overview 

As the grantee, Clemson University planned to adapt Project COFFEE (Cooperative 
Federation for Educational Experiences) in cooperation with three local school districts. Like 
the original COFFEE project developed in Massachusetts for youth with severe emotional 
disturbance, the Oconee County project was an abbreviated-day alternative school that 
integrated academic and vocational instruction to increase the likelihood that participants 
would complete school and be ready to enter the labor force in good entry-level jobs. The 
Oconee project, called OASIS, targeted highly at-risk students (not special education students, 
though some participants had been enrolled in special education during their educational 
experiences), many of whom were chronic truants with multiple suspensions, had been 
adjudicated for drug or alcohol-related offenses, had serious personal or family problems, and 
were seriously behind in their progress to graduation. Key features of the project included 
location in a separate facility, low student-teacher ratio, individualized instruction, clear and 
consistently enforced behavioral expectations, an entrepreneurial business (construction and 
sale of picnic furniture), and intensive psychosocial support for students. The project 
supplemented CDP funds with a large grant awarded by the South Carolina Department of 
Education. 

For Cohort 1, over two-thirds of treatment group members were male, compared with 
58 percent of Cohort 2 students. Nearly all participants were white (92 percent of Cohort 1 
and 83 percent of Cohort 2). Unlike the Academies or GRADS, substantially more OASIS 
students were overage for grade~58 percent of Cohort 1 and 92 percent of Cohort 2, a 
phenomenon that reflects their increased school problems relative to students of the other 
projects. Table B-4.1 in Appendix B provides comparable information for the site's control 
group. 

Plausible Outcomes 

Enrollment in OASIS could be expected to be associated with the following outcomes: 
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Improvement in school attendance aiid persistence 

Increase in self-esteem 

Increase in school affiliation 

Possible improvement in school performance. 



Results 



The Oconee County project, a random assignment site, began serving students in mid- 
March of school year 1989-90, which meant that students were exposed to the intervention for 
less than three months. A second randomly assigned cohort entered the alternative school in 
fall 1990, and the study collected information on both cohorts, as well as on randomly 
assigned control groups and a gap reduction group, through the 1991-1992 school year. 

Baseline to 1989-1990. The fact that most participants were very seriously deficient 
academically when they entered the project in spring 1990, combined with the typical 
■instructional winding down that occurs at the end of the academic year, helps to explain the 
lack of school performance outcomes among project participants. Analyses of data from 
Oconee County failed to show improvement in any of the academic performance areas for 
any of the analyses. 

On the other hand, the project's immediate success at facilitating participants' 
improved feelings about school and the school environment was reflected in several 
significant outcomes in the area of school affiliation. Participants viewed their teachers and 
instruction, and counselors and counseling, as better than they had previously experienced. 
The gap reduction analyses also showed a significant increase in the perception that high 
school prepared students to perform entry-level tasks in their chosen occupational field. The 
reliability-adjusted analysis of covariance showed an improved attitude toward teachers and 
teaching, while the gap reduction analysis showed an improved attitude toward counselors and 
counseling. 

Baseline to 1990-1991. Cohort 1 demonstrated improved school performance- 
measured as higher grade point average (from 1.9 to 2.7, versus a constant 2.1 for the control 
group)-on all analyses, although Cohort 2 did not achieve improvement in school 
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performance in comparison with the control group. Other findings included perception of 
better job preparation for Cohort 1 and a belief that school was safer among Cohort 2 
participants. Cohort I students, in comparison with those in the gap reduction group, 
perceived school as safer also. 

Baseline to 1991-1992 . Cohort 1 continued to improve their school performance, with 
significantly fewer courses failed and, compared with gap reduction students, fewer absences. 
While Cohort 2 did not show improvement in performance, the combined cohort for this 
school yeai- showed improvement in CPA and in number of credits earned. The only 
significant finding on nonperformance measures was a reduced perception that classmates 
should not misbehave for the combined cohort. 

DroDpine out . None of the OASIS cohorts for any year achieved significant effects 
for dropping out. In fact, in the second and third year for Cohort 1 and in both years for 
Cohort 2, very high numbers of participants and control group members dropped out of 
school (See Table B-4.4 in Appendix B). For example, by Cohort I's third year, 44 percent 
of participants and 58 percent of control group members had dropped out of or been expelled 
from school. Comparable figures for Cohort 2 were 70 percent for treatment, and 63 percent 
for control, group members and 46 percent and 44 percent for year two of the combined 
cohort. These high rates are in part explained by the fact that the project targeted highly at- 
risk youth, many of whom had previously dropped out or been expelled from school, and 
some of whom experienced legal and other problems that interfered with school attendance. 
Virtually all students were far behind their peers, and while participation in the project tended 
to improve their affiliation with school, for many this change was not sufficient to keep them 
in school. 



ERIC 



Summary 

OASIS represented a "last chance" for many of its participants. Eariy on, the school 
was characterized by a high level of caring and commitment on the part of staff, many of 
whom spent long hours working on strategies for improving their students' educational 
experiences. Additionally, many staff, in particular the school's first director, worked closely 
with students, parents, and others in attempts to solve students' personal, social, and legal 
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problems. His level of commitment to the school and the project's activities, the small 
classes, the student-operated business, the highly committed staff, and the personalization of 
services initially worked well for many students. However, some factors in the second year 
tended to cause problems for OASIS students, in particular the district's retraction of waivers 
regarding physical education and students' smoking on campus. Additionally, the stigma 
associated with attending the school was difficult for many students, as were the negative 
attitudes of other students experienced by OASIS students who attended the area vocational 
school during 1991-1992. Finally, the departure of the director at the end of the project's 
second year may have been discouraging to some of the students with whom he had worked 
closely. At the same time, the project did demonstrate some positive outcomes, and many of 
its participants became substantially more affiliated with educational settings and experiences 
than had previously been the case. 



Youth Experiencing Success (Y.E.S.) 
Center for Applied Technologv South. Anne Arundel County. Maryland 

Overview 

Intended by the grantee, Clemson University, to be an adaptation of COFFEE, the 
Anne Arundel County project was unable to implement the model as intended, primarily 
because of lack of sufficient funds to support a replication or adaptation. Hence, rather than 
establishing an alternative school in a separate location, the project operated in one of the 
school district's area vocational-technical centers. Students took most academic courses at 
their home high schools and vocational courses at the center, which they attended for half of 
each day. Services provided to students during the first year (beginning in the second 
semester) included (1) enrollment in a computer-based, career-related English course whose 
emphasis was on communication skills, and (2) support from vocational specialists in the 
school's shops and labs, whose responsibility was to work with individual project participants 
during the regular class period on components of their lessons with which they were having 
difficulty. These two activities continued in subsequent years, and students were also offered 
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the opportunity for credit-bearing community-based education placements. The latter were 
intended to relate to students' vocational programs, although lack of available jobs meant that 
some students were unable to work in related areas. 

Most project participants were male (81 percent of Cohort 1 and 72 percent of 
Cohort 2). A majority were white (52 and 61 percent); for the second cohort, 39 percent of 
participants were black. Eighty-one percent of Cohort 1 students were overage for grade, 
compared with 50 percent of students in Cohort 2. Table B-5.1 in Appendix B provides 
comparable information for the site's control group members. 

Plausible Outcomes 

Among the outcomes that might result from project participation are: 

• Improvement in vocational course grades 

• Improvement in communication skills and possible improvement in grades and 
number of credits earned in English. 

Results 

While Anne Arundel, a random assignment site, began serving students in January 
1990, the full complex of services was not available until the 1990-91 school year. A second 
cohort was selected in fall 1990, and both cohorts were followed through the end of the 1991- 
1992 school year. 

Baseline to 1989-1990 . The fact that students participated in the project for one 
semester only during their first year may account for the fact that outcome data for this 
project showed only three statisticpJly significant results. The reliability-adjusted covariance 
analysis showed a significant decrease in the number of days absent relative to the control 
group, although the average number of days absent actually increased from 9.51 in the spring 
semester of the year prior to the treatment to 1 1 .87 days in the spring semester of the first 
year of treatment (the control group's mean number of days absent increased from 7.00 to 
13.98). The gap reduction analyses found a significant improvement in grade point average 
(from .99 to 1.34). 
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Baseline to 1990-1 99 L On the basis of the gap reduction analyses. Cohort 1 students 
continued to show improvement in school performance as measured by grade point average. 
Additionally, Cohort 1 participants had an increasing perception that students in their classes 
were preparing for college and that they were receiving better job preparation. There were no 
significant results for Cohort 2 during this year. 

Baseline to 1991-1992. Cohort 1 experienced improvement in grade point outcomes in 
comparison with the gap reduction group; Cohort 2 experienced the same improvement on 
gain score analyses (with a change from 1.3 to 2.3, in comparison with 1.2 to 1.8 for the 
control group), as did the combined cohort for year two. Cohort 2 and the combined cohort 
perceived school as safer, also. 

Dropping out. No significant results occurred in terms of dropping out. Both 
treatment and control groups experienced about the same rates of dropping out: 47 percent 
for Cohort 1 treatment group members and 48 percent for control group members in their 
third year (school year 1991-1992); and 22 percent for both Cohort 2 groups in their second 
year. For the combined cohort, the year two treatment group lost 13 of 37 participants, while 
the control group lost 14 of 41 (35 and 34 percent, respectively). 

Summary 

Unlike the Oconee County alternative school, Anne Arundel was unable to obtain 
supplementary funds to support implementation of the COFFEE model that the grantee 
proposed to implement. Thus, the total resources available to the site were relatively small, a 
factor that affected the types and intensity of services that the project was able to offer 
participants. The fact that participants improved in grade point averages is probably directly 
attributable to two of the project's services: (1) the availability of a credit-bearing English 
course at the vocational-technical school, which emphasized communication and other skills 
that were closer to vocational students' interests than more traditionally "academic" English; 
and (2) students' participation in community-based education, which gave them credit-and 
pay-tor jobs that related to their vocational interests. In addition to limited resources, a 
relative lack of school and district-level support for the project may have reduced its 
cllcctivcncss with students. At the same time, students we interviewed expressed consistently 
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positive attitudes toward some project activities, in particular the assistance they received in 
completing their vocational coursework and the community-based work experience in which 
they participated. 



Changing How Our Pupils Succeed fC.H.O.P.S.) 
McFatter Vocational-Technical Center. Broward County. Florida 

Overview 

The third of the local project sites implemented under the gr. nt to Clemson 
University, the C.H.O.P.S. project operated in an area vocational-technical school under the 
administrative purview of the school's director. Students took their academic courses at the 
vocational-technical center, entirely through computer-assisted instruction (CAI). (While the 
project staff intended to implement applied academics that integrated academic with 
vocational instruction, the school's director selected the CAI model instead.) Project 
participants enrolled in vocational programs at the center and were eligible to participate in 
the center's Industrial/Community-based Education (ICE) program, which permitted students 
to work in the community once they had completed 50 percent of their vocational 
competencies and, while working, to attend school only one day a week. Project components 
included counseling and other support activities, competency-based curriculum, student 
contracts, and a nonpunitive approach to attendance and discipline. 

Initiated in fall 1990, the project involved only one study cohort. Most participants 
(85 percent) were male; 46 percent were white, 25 percent Native American, and 21 percent 
black. Nearly all (88 percent) were overage for grade, one of the criteria for enrollment in 
the project. Table B-6.1 in Appendix B provides comparable information on the site's control 
group members. 

Plausible Outcomes 

The project's participants, given enrollment in and completion of C.H.O.P.S., could be 
expected to achieve the following outcomes: 
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Improvement in school persistence 

Improvement in postschool employment prospects, based on completion of a 
vocational program. ^ 



Results 



The Broward project began operations in fall 1990; individuals were randomly 
assigned to treatment and control groups for the study's one cohort. Additionally, the site 
established a gap reduction group. The evaluation experienced a number of administrative 
delays and difficulties, centering around arrangement for implementing the design. District 
administrative stmctures meant that the project was located in an area vocational-technical 
center in the district's vocational education division, while the project itself was under the 
purview of a project coordinator who reported to the district official in charge of special 
needs. Because vocational-technical directors have considerable autonomy, implementation of 
the project within the center was largely dictated by the director rather than by the project 
coordinator. In addition, the evaluation staff experienced delays in obtaining local permission 
for the study, although the grantee had agreed to participate. These issues and delays affected 
the evaluation's ability to collect information and track students. 

MsdinejoJ990dm. Gap reduction analyses showed a negative result among 
project participants in terms of number of credits earned. No other outcomes were significant 
in either direction. 

MelinejoJ99U992. Gap reduction analyses showed a negative result among 
project participants in terms of number of credits eamed. No other outcomes were significant 
in either direction. 

^'"^^'"^ ^" y^^' ^hree participants (16 percent) and four control group 

students (25 percent) dropped out. By the end of the second year, six (32 percent) 
participants and seven (44 percent) controls dropped out. None of these results was 
significant. 
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Summary 

Because of a variety of implementation problems, both with the project itself and with 
the evaluation, results in this site were disappointing. Coordination among project staff and 
administrators was limited, and the site's version of applied academics was restricted to CAI. 
Even so, evaluation staff noted that the project coordinator and teaching staff had been 
effective in creating a nurturing enviionment for participants, including availability of 
personal counseling, a nonpunitive approach to attendance and discipline, and availability of 
tutoring support for students requesting it. 

Vocational Continuum Program 
Grant Higii School. Portland, Oregon 

Overview 

The Multnomah County School District No. 1, serving Portland, Oregon, obtained 
CD? funding to expand the district's Vocational Continuum Program, a multicomponent 
program whose overall purpose was to provide an integrated continuum of vocational support 
services for the district's at-risk students. Components of the program included the BRIDGE 
program for ninth and tenth graders, a curriculum intended to facilitate students' transition 
from middle to high school through provision of prevocational instruction, tutorial assistance, 
school socialization, study skills and time management, and other supports. Counseling and 
case management services were also available as appropriate; students participated in 
BRIDGE cia.ss for one hour each day. Following BRIDGE, students were to enter one of 
three options, including mainstream vocational technical education, applied 
academics/cooperative work experience, and Vocational Mentoring based m the experience- 
based career education model. Vocational Advocates would provide ongoing vocational 
counseling and guidance. Participants, students enrolled at Grant High School, were to spend 
one-half of each day at a local hospital for exploration of health-related careers and applied 
academics. 
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Because the project's timelines began with students recruited into BRIDGE in school 
year 1989-1990, the evaluation included these students in the first year and then selected a 
cohort of Grant High School students enrolled in Vocational Mentoring and matched 
comparison students as a second cohort beginning in school year 1990-1991. While some of 
the latter cohort had been in BRIDGE, not all had received the earlier component of the 
continuum. 

Across the two study cohorts. Continuum students differed somewhat in terms of 
demographic characteristics. For example. Cohort 1 was predominantly male (78 percent), 
while a majority of Cohort 2 students were female (54 percent). Most Cohort 1 students were 
black (87 percent), compared with 46 percent of those in Cohort 2, which was 50 percent 
white. Finally, a majority of Cohort 1 students were overage for grade (52 percent), versus 
38 percent of students in the second cohort. Table B-7.1 in Appendix B provides 
demographic information for the site's comparison group members. 

Plausible Outcomes 

Outcomes that could be expected to result from participation in BRIDGE include the 
following: 

• Increase in school affiliation 

• Possible increase in retention 

• Eventual improvement in school performance. 

Outcomes that could be expected of Grant's Vocational Mentoring students in eleventh and 
twelfth grades include: 

• Improved school retention 

• Improved postschool employment options. 

Results 

Portland was a matched comparison group site. As noted in the overview, the 
evaluation selected two cohorts, one from BRIDGE and one from Vocational Mentoring, 
along with matched comparison groups and a gap reduction group. 
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Baseline to 1989-1990 . Enrollment in BRIDGE classes one hour per day had the 
general purpose of improving students' transition to high school, and the measurable benefit 
of this experience, with a fairly diverse set of activities for students characterized by a fairly 
high level of individualization, might require a longer term to demonstrate performance or 
affiliation effects. That participation had some effect on students' view of their educational 
experience is suggested by the finding that they perceived their counselors and counseling 
services as better. More specifically, the reliability-adjusted analysis of covariance showed a 
significant improvement in student attitudes towards counselors and counseling. In addition, 
Portland showed a significant reduction in the number of credits earned both in the regular 
analysis of covariance and the gap reduction analysis among Cohort 1 students. 

Baseline to 1990-1991 . Cohort 1 students failed fewer courses during their second 
year, while Cohort 2 students increased their absences, relative to the comparison group. 
Cohort 1 students perceived school as safer and believed that classmates were college bound, 
while Cohort 2 students demonstrated improved attitudes toward teachers and teaching and 
believed that their future expectations were better. The latter, unlike Cohort 1, experienced 
reduced perceptions that classmates were college bound. The combined cohort believed that 
school was preparing them for entry-level jobs. 

Baseline to 1991-1992 . Both cohorts improved in the number of credits earned, and 
Cohort 2 experienced a reduction in number of absences, relative to the comparison group. 
Gap reduction analyses indicated an increased number of credits and fewer courses failed for 
Cohort 1 and higher GPA and credits earned for Cohort 2. Cohort 1 perceived increased 
academic encouragement but less positive future expectations, and Cohort 2 thought school 
was safer. The combined cohort was positive on all school performance measures except 
dropping out, including higher GPA (from 1.9 to 2.0, versus 2.2 to 1.8 for the comparison 
group), higher number of credits, fewer absences, and fewer courses failed (1.7 to .7 for 
participants, 1.4 to 1.6 for comparison students). 

Dropping out . Analysis of data from this site revealed no significant differences in 
dropping out across years or across cohorts. As shown in Table B-7.4 (Appendix B), 
substantial numbers of students dropped out or were expelled. For Cohort 1, by the end of 
the third year, 30 percent of the treatment group and 17 percent of the comparison group had 
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dropped out. For Cohort 2, 24 percent of the treatment group and 31 percent of the 
comparison group were no longer in school. Corresponding figures for the combined cohort 
were 23 and 26 percent, respectively, at the end of year 2. 

Summary 

Students recruited for the district's Vocational Continuum were typically highly at 
risk. Proportionally more students at the targeted high school were minority, eligible for free 
or reduced-price lunch, and more likely to drop out of school (the district's rate is between 25 
to 30 percent, while the high school's was over 30 percent). Key components of the 
continuum intended to address the specific needs of such students, including individualized 
treatment, coordinated case management, mentoring experiences that paired students with 
hospital employees, small classes, and work experience opportunities. That services received 
by participants followed in the evaluation had a salutary effect is suggested by the generally 
positive outcomes achieved. While the project did not significantly affect dropping out, it did 
facilitate improved school performance among participants as well as improved school 
affiliation and generally more positive identification with school. These findings suggest that 
the project, particularly the components provided to students in Vocational Mentoring, was 
effective in reaching participants. 



Preparing At-Risk Youth for Employment Program 
Breithaupt Vocational Technical Center, Detroit, Michigan 



Overview 



Called STARS (Support Team for At Risk Students), the Detroit project operated in 
four of the district's area vocational-technical centers. The project provided support services 
to at-risk students enrolled in the vocational-technical centers, which operated on a two-shift 
schedule with vocational students spending one-half of each day pursuing a vocational 
program and the other half at their home high schools taking academic classes. Project staff 
served as vocational specialists, assisting students in the center's shops and labs with their 
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technical courses; staff also provided counseling support and coordination with students' 
home high schools. Finally, the project employed advocates who assisted limited English 
proficient participants with their coursework. The evaluation selected students from one of 
the area centers for tracking across two years of project participation. 

Most students in the cohort were male (70 perceut) and black (87 percent), with five 
percent white and seven percent Hispanic. Nearly all (79 percent) were overage for grade. 
Table B-8.1 provides comparable information for control group members. 



Plausible Outcomes 

Completion of the project could be expected to result in the following outcomes for 

participants: 

• Improved vocational competencies 

• Somewhat increased school retention 

• Improved postschool employment options. 



Results 

Owing to major changes in the district's educational leadership and organization in fall 
1989, the project did not get underway until spring 1990. Consequently, the evaluation, using 
a random assignment design, sekcted only one cohort, including treatment, control, and gap 
reduction groups, in fall 1990. While the project operated in several area vocational-technical 
centers, the evaluation limited data collection to students attending Breithaupt, a 
comprehensive area vocational-technical school known for vocational programs in meat 
handling, training of students to be chefs, and food service. The school has a widely regarded 
restaurant, where students learned the full range of food preparation and service. We 
followed participating and control students over two school years, through spring of 1992. 

Baseline to 1989-1990 . The project did not enroll a cohort in fall 1989 owing to 
delays in implementation. 

Baseline to 1990-1991 . During the first year, treatment students had higher grade 
point averages and earned a higher number of credits than did control group members. Both 
groups experienced a decline in credits, with the control group's decline greater (treatment 
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group decline of 28,6 to 21.8 versus 28.1 to 17.4 for the control group). In comparison with 
gap reduction students, participants earned higher GPAs but also had more absences. There 
were no significant school affiliation or student perception outcomes for this year, 

Baseline to 1991-1992 . Treatment group students continued to achieve a higher GPA 
compared with the control and gap reduction groups; they had more absences than the gap 
reduction group but also failed fewer courses. Their perceptions of positive future 
expectations declined relative to the control group. 

Dropping out . The project achieved a significant reduction in dropping out. For 
1990-1991, only three of 87 treatment students dropped out, compared to 13 of 96 control 
group members. For 1991-1992, six more of the participant group dropped out, versus 12 
more of the control group. The effect was greatest among overage students: 24 of 65 
overage control group students dropped out, while only seven of 69 overage participants 
dropped out. Dropout rates by the end of the second year were 1 1 percent for the treatment 
group and 26 percent for the control group. 

Summary 

Following a number of delays owing to changes in the district's administration and 
restructuring of the subdistricts in Detroit, the project began serving students in late spring 
1990. To facilitate recruitment, the project was able to obtain a waiver from the district 
policy that created disincentives for students to attend the v.ocational centers (having to do 
with student-teacher ratios and consequent potential loss of teachers at the comprehensive 
high schools when students enrolled in the vocational schools). Once underway, the project 
provided substantial support services for identified students at the center, which is a well- 
equipped and well-managed vocational training facility.^ Our findings suggest that the 
project succeeded in keeping participants in school as well as facilitating improvements in 
school performance. 



-'Detroit's five area centers were constructed as part of a desegregation initiative; all are well equipped, though 
none is apparently operating to full student capacity. As in other large cities, many students (and parents) perceive 
vocational education to he unappealing, and stereotypically for "dumb" students. 
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Project COFFEE Vocational Training Program 



Turtle Mountain. North Dakota 



The grantee, the North Dakota Board of Vocational Education, intended to replicate 
Project COFFEE, working with high schools in or near the state's four Indian Reservations. 
The intent was to work with all high schools in each area (typically three, though in one site 
there were only two and in another site one of the three schools declined to participate). In 
general, for a variety of logistical, resource, and political reasons, the sites were not able to 
replicate the model but rather adapted it to local conditions in response to various constraints. 
Overall, the four North Dakota sites implemented a similar design that included several key 
components: (1) employment of an academic instructor to provide academic tutoring in 
subjects with which students were having difficulty, with the instruction provided on a pull- 
out basis; (2) employment of a vocational instructor to provide career guidance and 
employability skills instruction and to arrange work experience slots for students to the extent 
possible; and (3) payment of students for school attendance. The projects began serving 
students in mid-March 1989, which meant that participation in the first year was limited. 

Overview 

The Turtle Mountain site drew students from two area high schools. In its first two 
years, the site was more successful than others in efforts to implement a project comprising 
some of the key features of Project COFFEE. The project was located in a separate facility 
and served primarily dropouts who were working toward a GED. The academic instructor 
was working on integrating elements of the students' native culture into the career 
development curriculum and hoped to find participants work experience that would involve 
recording of their tribe's oral history. The vocational instructor, who had enrolled students in 
courses at an area vocational-technical center, was called into active military duty for the Gulf 
War, leaving the project with only one instructor for most of school yeai' 1990-1991. 

The majority of project participants in both cohorts was male (56 percent and 55 
percent). AH were Native American, and nearly all (88 percent and 82 percent) were overage 
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for grade. Table B-9.1 in Appendix B provides comparable information for the site's control 
group members. 



Plausible Outcomes 

Among the outcomes that this project might be expected to achieve are: 

• Improvement in school attendance 

• Improvement in school affiliation 

• Increase in number of GEDs awarded. 

Results 

The Turtle Mountain site implemented random assignment, with selection of 
treatment and control groups in spring 1990 as the project got underway and again in fall 
1990. Both cohorts also had a gap reduction group. Participants were drawn from two area 
schools; a third school, located on the reservation, declined to participate in the project. As 
with other study sites with two cohorts, the evaluation followed the first cohort for three years 
and the second for two years. Results include analyses for both cohorts and for the combined 
cohort's year 1 and year 2 experiences. 

Baseline to 1989-1990. Students enrolled in the COFFEE project at Turtle Mountain 
improved their school performance in the area of attendance. In fact. Turtle Mountain, a 
random assignment site, is one of the few sites that showed a significant reduction in the 
number of days absent in both the reliability-adjusted and the regular analyses of covariance. 
Factors that may help to explain this finding include their receipt of payment for attendance 
and, perhaps more important, the commitment of the academic instructor to ensuring students' 
attendance. This commitment was reflected in her routine practice of picking participants up 
at their homes and driving them to school. Both covariance analyses indicated reductions in 
future expectations. As with the other North Dakota sites, it is important to remember that 
projects began serving students in March, leaving little time in the first year for interventions 
to demonstrate much in the way of student outcomes. 



ERIC 



4-34 



118 



Baseline to 1990-1991 . Other than dropping out, none of the school performance 
outcomes was significant during the year for Cohort 1. Cohort 2 students, in comparison with 
the gap reduction group, had higher grade point averages and failed fewer courses. The 
combined cohort had higher CPA. The combined cohort, versus the control group, felt safer 
at school. 

Baseline to 1991-1992 . None of the treatment-control or gap reduction comparisons 
met the criteria for significance for this year. 

Dropping out . Turtle Mountain's Cohort 1 achieved significant, positive effects in 
dropping out for both 1989-1990 and 1990-1991, and the combined cohort had significant 
effects for year 1. Fifteen of 19 control group students dropped out by the end of the second 
year, compared with five of 15 treatment group students. By the end of their third year, 
however, an additional five treatment group students dropped out, for a 67 percent dropout 
rate, compared with 84 percent of control group students. Cohort 2's second-year rates were 
50 percent for treatment, and 38 percent for control group members, and the combined 
cohort's second-year rates were 40 and 63 percent, respectively. 

Summary 

Projects implemented in North Dakota faced a number of environmental and 
implementation-related challenges that probably constrained their success in implementing the 
intended model and achieving positive student outcomes. Within these constraints, however, 
the Turtle Mountain project was able to have a significant effect on participants' retention, at 
least through their second year of participation. Additionally, the project had a positive effect 
on attendance in the first year for Cohort 1, possibly related to the policy of paying students 
for attendance. Given the difficulties faced by the project, these positive findings suggest the 
effectiveness of the work of the project's academic teacher, whose commitment and creativity 
were key to the progress the site was able to make. 
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Proiect COFFEE Vocational Training Program 



Fort Totten. North Dakota 



Overview 

As with the other sites in North Dakota, resource and environmental constraints 
required the project to implement a limited version of the intended project model. The 
project operated at two high schools, one on the reservation and one nearby, with a project 
staff member (officially instructors but also serving as counselors and student advocates) 
located in each school. These instructors provided academic remediation on an individualized 
basis to project participants. Additionally, the instructor located at one of the two 
participating schools provided considerable individual counseling and psychosocial support to 
project participants. In the first year, some participants enrolled in vocational classes at an 
area vocational-technical center and received vocational counseling and employability skills 
instruction from project staff. Students received payment for school attendance. 

A majority of participants in the two treatment cohorts were male (61 percent of 
Cohort 1 and 87 percent of Cohort 2). All Cohort 1 participants were Native American, 
while one of the 15 Cohort 2 participants was white and the remainder Native American. 
Most were overage for grade--78 percent of Cohort 1 and 80 percent of Cohort 2. Table B- 
10.1 provides demographic information on the control groups. 

Plausible Outcomes 

Among the outcomes that might result from participation are: 

• Improvement in attendance 

• Possible improvement in school performance. 
Results 

The project used random assignment for the two cohorts and control group members 
and also selected a gap reduction group for each of the two cohorts. As with other North 
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Dakota sites, services began in March 1990, which meant that the Cohort 1 students had 
limited access to project services during the first year. 

Baseline to 1989-1990 . Even within the limited time frame of the intervention during 
the 1989-90 school year. Fort Totten showed school performance gains: both the reliability- 
adjusted covariance analyses and the gap reduction analyses showed a reduction in the 
number of courses failed and an increase in grade point average, although students also 
earned fewer credits. Conversely, the site showed a significant negative treatment effect on 
the number of credits earned in both analyses of covariance. The two analyses of covariance 
were also consistent in finding a worsening attitude toward counselors and counseling. 

Baseline to 1990-1991 . Cohort 1 showed significant outcomes on gap reduction 
analyses for higher CPA and fewer courses failed. The cohort also perceived improved job 
preparation compared with the gap reduction group. Cohort 2 achieved higher CPA in 
comparison with the control and gap reduction groups, as did the combined cohort for this 
year. 

Baseline to 1991-1992 . The combined cohort, for the second year of participation, 
experienced an increase in the number of courses failed versus the control group and a higher 
CPA in comparison to the gap reduction group. The combined cohort also believed that 
classmates should not misbehave, based on control group comparisons. 

Dropping out . Dropout performance was worse for the treatment group for Cohort I's 
first year (1989- 1990) but significantly better for Cohort 2's second year (1991-1992), when 
one of 14 participating students dropped out (versus 18 of 43 control group members). By 
the end of the third year, most Cohort 1 students had dropped out (74 percent), versus 41 
percent of control group members. As noted earlier. Cohort 2 performed better on this 
outcome (seven versus 42 percent). 



Summary 

Given the challenges faced by the project, the presence of positive results on some 
school performance measures suggests that the additional support provided to participants by 
the instructors was important. At one school in particular, the instructor provided a great deal 
of academic and personal support to participants, advising them on their programs, helping 
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them to work out personal problems, and mentoring them as they began the transition from 
school. Further, receipt of payment for attendance likely motivated some students to remain ' 
in school. Another factor, mentioned by instructors and others we interviewed, was the 
importance of free meals that students received in school; for some, these were the only 
meals of their day. 



Project COFFEE Vocational Training Program 
Fort Berthold. North Dakota 

Overview 

Like the other North Dakota sites. Fort Berthold employed two instructors, one 
academic and one vocational, who provided services to participants who attended three widely 
dispersed schools in the area. During the first year of the project, the instructors were located 
in a retrofitted trailer on the campus of the area vocational-technical school in one locality, 
and students from the three participating schools were bused once a week to the trailer for 
academic remediation and career guidance and employability development. Primarily because 
of distances (one school was 60 miles away and another 35 miles distant), participation rates 
were very low, with only students from the nearby high school attending with any regularity. 
For the second and third years, instructors "rode circuit," visiting each of the schools during 
the week rather than asking students to travel to them. As with the other sites, students were 
paid for school attendance. 

Most project participants were male (62 percent of Cohort 1 and 67 percent of 
Cohort 2); nearly ail were Native American (with Cohort 2 enrolling one white student). A 
substantial majority were overage for grade--69 percent of Cohort 1 and 73 percent of 
Cohort 2. Table B-11.1 provides information on characteristics of the control group. 
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Plausible Outcomes 

Project participants could be expected to make gains in the following areas: 

• Improvement in attendance 

• Possible improvement in school performance. 
Results 

Fort Berthold, North Dakota, a random assignment site, enrolled two cohorts, the first 
selected in spring 1990 and the second for fall 1990. Each cohort included a randomly 
assigned control group and a gap reduction group. 

Baseline to 1989-1990 . Treatment group members in Cohort 1 showed a significant 
increase in the number of courses failed in the gap reduction analysis. Both analyses of 
covariance found a reduced perception of receiving preparation for entry-level tasks in 
students' chosen occupation and an increased sensitivity toward the disruptive behavior of 
their classmates. The reliability-adjusted covariance analysis and the gap reduction analysis 
both found an increasing perception that classmates were preparing for college. The regular 
analysis of covariance found reduced expectations for the future. Finally, gap reduction 
analyses found a reduced perception among participants of receiving adequate help in 
choosing courses. 

As with other project sites, project services at Fort Berthold were available to students 
for a short period in the spring of the year, a reality that must be considered in viewing these 
results. Further, the project's service arrangement for this period required that students be 
bused long distances from two of the three participating schools to receive services. Students 
often chose not tc make the trip. Recognizing this problem, the project revised the service 
plan for year two, taking the services to the students. School administrators and project staff 
were considerably more comfortable with this arrangement. 

Baseline to 1990-1991 . By the end of this year, most Cohort 1 students were no 
longer available for analysis, and no significant results were found. For Cohort 2, participants 
earned fewer credits in comparison with the control group but more than the gap reduction 
group, as did the combined cohort. Cohort 2 students perceived teachers and teaching, and 
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counselors and counseling, more negatively than did the control group, but did perceive the 
school as safer and believed that classmates should not misbehave. 

Baseline to 1991-1992 . For the combined cohort, treatment students had fewer 
absences and failed fewer courses than did the control group. They earned more credits in 
comparison with the gap reduction group. Cohort 2 students continued to perceive teachers 
and teaching as worse. 

Dropping out . Performance on this outcome was not significant, although dropout 
rates were quite high. For example, by the third year, 60 percent of Cohort 1 treatment 
students and 40 percent of control students had dropped out, compared with only 14 percent 
of Cohort 2 treatment, and 21 percent of control group members. However, the numbers 
available for analysis were consistently small in this site. In addition to students verified as 
having dropped out, the evaluation was unable to track most students after the first year, 
which seriously limits confidence in the findings. 

Summary 

The distances among the three schools participating in the project, along with 
relatively low levels of administrative support in two of the three schools, limited the ability 
of the project to perform. While the two project-employed instructors were enthusiastic and 
concerned, the necessity to implement an essentially pull-out remediation model of services, 
with low levels of integration between academic and vocational components, combined with 
limited vocational offerings and limited job opportunities, constrained their ability to improve 
the educational experiences and outcomes of participants. 



4-40 

124 



Prolect COFFEE Vocational Training Program 
Fort Yates (Standing Rock). North Dakota 



Overview 

Like other North Dakota sites, a variety of factors, including resource limitations, 
weak administrative support at some schools, and lack of work experience opportunities, 
prevented the site from implementing a COFFEE model. The project employed an academic 
instructor who provided individualized instruction on a pull-out basis to students who were 
having difficulty in their academic classes and a vocational instructor who provided career 
guidance and employability skills instruction. Additionally, the vocational instructor 
attempted to develop work experience slots for participants, although the very high local 
unemployment rate (nearly 80 percent) limited effectiveness of this effort. Students, who 
were drawn from three area schools, received payment for school attendance. 

Most participants were male~53 percent of Cohort 1 and 69 percent of Cohort 2, and 
nearly all were Native American. A substantial majority were overage for grade-76 percent 
of Cohort 1 and 63 percent of Cohort 2. Information on characteristics of control group 
members appears in Table B-13.1 in Appendix B. 

Plausible Outcomes 

Expectable outcomes include the following: 

• Improvement in school attendance 

• Possible improvement in school performance. 
Results 

Like the other sites. Fort Yates implemented a random assignment design along with a 
gap reduction group. The first cohort entered the project in spring 1990, and the second in 
fall 1990. 

Baseline to 1989-1990 . In its first year, the project at Fort Yates did not demonstrate 
significant outcomes for any of the measures of interest. This project site was located in an 
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area characterized by the most difficult environmental problems of any in the evaluation. 
Unemployment was at 79 percent; there were almost no jobs available. Per capita income 
was very low, and many students were reported by school and project staff to come to school 
at least in part to eat breakfast and lunch--the only meals of the day for many of them. It 
was difficult to attract teachers, and relationships ^tween the two public school districts and 
the BIA school were problematic. Furtlier, many of the school administrators did not support 
the project, which made implementation even more difficult. The project's director hired two 
highly experienced and committed instructors for this site, and hoped that she and they would 
be able to solve some of the problems that this site experienced during the first year. 

Baseline to 1990-1991 . Cohort 1 achieved no significant school performance 
outcomes. Further, participants perceived teachers and teaching to be worse and that they 
received less academic encouragement in comparison with the control group. Cohort 2 
experienced higher grade point averages (1.8 to 2.0 versus the control group's 1.9 to 1.5). 
The combined cohort also had a significantly higher grade point average. Cohort 2 had fewer 
absences than the control group; both groups' absences increased--the control group's from 
eight to 13 days and the treatment group's from eight to 10. 

Baseline to 1991-1992 . Neither cohort achieved significant improvement in any 
school performance outcomes, and the combined cohort failed significantly more courses. 
Similarly, there were no significant findings in school affiliation or student perception 
outcomes, other than Cohort 2 (and the combined cohort's) view that many of their 
classmates were college bound. 

Dropping out . While the results were nonsignificant, substantially fewer students 
dropped out than was the case at the other North Dakota sites. For Cohort 1, only 19 percent 
of treatment students and 3 1 percent of controls had dropped out by the end of the third year, 
compared with 19 percent of treatment and 23 percent of control students in Cohort 2. These 
findings may reflect the severe restriction on nonschool options (particularly lack of jobs) 
available to youth in the Standing Rock area. 
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Summary 

As noted earlier, environmental conditions in the area were consistently difficult. This 
problem was exacerbated by the lack of support for the project among school administrators, 
particularly those at the BIA school. At the same time, the fact that students received 
payment for school attendance probably had a salutary effect on their retention. Nevertheless, 
substantially more intensive interventions would likely be necessary to improve the 
educational experiences and outcomes of the youth living on or near this reservation. 
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CHAPTER 5: SUMMARY AND IMPLICATIONS 



As noted throughout this report on the experience of the CDF dropout prevention and 
reentry projects, a number of factors affected the extent to which each of the projects 
included in the in-depth evaluation succeeded in implementing the project designs for which 
they received funding from OVAE and in achieving positive effects on participants' school 
performance and affiliation. Many of the early problems experienced by the projects resulted 
from issues of timing, in that the grantees received funding during the summer, too late to 
complete planning, staffing, and logistical arrangements required to begin serving students 
according to their initial schedules in fall of the 1989-90 school year. Two of the sites were 
unable to initiate services at all during the 1989-90 school year, while others began serving 
students during the second semester. To some extent the first year of the demonstration was 
really a "pilot" for these projects, during which they worked out operational details and 
essentially prepared for full-scale implementation in school year 1990-91. The few projects 
that began serving students in September 1989 experienced delays that affected some 
components of their activities. For example. Gushing, Woodside, and Carlmont were unable 
to get their computer labs underway until mid-year, primarily because of delays in delivery 
and installation of equipment. In some sense, then, evaluation of the second and third years 
of the projects is a fairer test of the types of outcomes that dropout prevention projects using 
vocational education as an intervention can achieve with students. 

The relatively modest student-related outcomes demonstrated by many of the projects 
over the course of the evaluation reflect the difficulty of overcoming the myriad political, 
logistical, and environmental factors that require attention as grantees attempt to implement 
even well-conceived and tested strategies for improving the educational experiences and 
outcomes of at-risk adolescents. Perhaps more important, they reflect the difficulty of 
changing what have typically been many years of unproductive-often painful-educational 
experiences for these youth. Even so, the demonstration included some successful projects, 
reflecting a variety of strategies for addressing the problems of their target population. These 
strategies included the school-within-a-school model of the two Business Technology 
Academies, the alternative school f' r highly at-risk youth as implemented in rural South 



Carolina, and the supportive services provided to students attending vocational-technical 
schools in both rural (Gushing) and urban (Detroit) settings. In this chapter we discuss oi 
findings regarding the relative success of the projects in the evaluation. We begin with a 
brief review of the projects' outcomes as a context for our discussion of some possible 
explanations for the findings. 



Summary of Participant Outcomes 

As noted throughout this report, the CDP demonstration's primary intended outcome 
was reduction in dropping out among project participants, or, for reentry projects, return to 
school. Given the likelihood that projects might achieve other, intermediate, outcomes (e.g., 
improvement in school affiliation or performance) prior to changes in dropping out, we also 
examined a number of these outcomes. In sum, the number of the 12 total project sites that 
achieved each of the outcomes of interest, relative to the outcomes of the comparison or 
control group, in the evaluation were as follows. 

Outcome Number of Projects With Outcome 

• Reduction in dropping out 4 

• Increase in grade point average 10 

• Reduction in number of courses failed 7 

• Increase in number of credits earned 5 

• Reduction in number of absences 5 



Improvement in students' perception of 
teachers and instruction 

Improvement in students' perception of 
counselors and counseling 
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• Increase in students' perception that 

school is safe 7 

• Students' perception of receiving more academic 
encouragement ^ 

• Students' perception of receiving better job 
preparation 3 

As this summary notes, only one-third of the projects achieved significant reduction in 

the number of participants who dropped out of school. They were generally more successful, 

however, in terms of school performance and affiliation, with 10 of the 12 demonstrating 

increases in participants' grade point average, seven showing reduction in number of courses 

failed, and seven showing improvements in participants' perceptions of the safety of their 

school environment. Five projects achieved reduction in the number of absences. Subsequent 

sections of this chapter discuss these findings in the context of the projects' "treatment" and 

relative levels of implementation over the three-year life of the demonstration. 



Proximity 

Our experience with the projects that participated in the evaluation suggest that 
organizational location, in combination with the location of project sites and services, affected 
project implementation and consequently participant outcomes. Three of the grantees 
(Sequoia Union High School District, Detroit Public Schools, and School District #1, 
Portland, Oregon) were local education agencies, with grantee staff located in district 
administrative offices. Sequoia and Portland located the project in comprehensive high 
schools, while Detroit provided services to students attending four of the district's five area 
vocational-technical centers. While school-level support was a problem for one or two of 
these project sites, in general the organizational (and probably geographical) proximity of the 
grantee to the location of services increased the likelihood that project staff could stay on top 
of project activities. In particular, this proximity facilitated identification of appropriate staff, 
resolution of resource problems and needs, and ongoing "hand holding." The fact that project 

5-3 

o 130 
ERIC 



directors were central office staff carried with it a certain credibility that also facilitated 
cooperation at the school level. While other factors contributed to these projects' success, 
location of the project within the administrative structure that governed the location of project 
services probably contributed to the efficiency and effectiveness of the projects. 

Conversely, three other grantees were physically and administratively distant from the 
sites. The Gushing project, located in a regional educational service organization with the 
project in a nearby area vocational-technical center (with 1 1 LEAs feeding students to the 
center), faced challenges in developing good working relationships with the school's staff and 
administrators, in part because the project staff were initially viewed as "outsiders" who 
lacked understanding of the area school's informal structure and were not widely experienced 
in vocational education. The fact that project staff all possessed extensive experience in 
working with school and district teachers and administrators around the state and nation in 
their role as facilitators for the National Diffusion Network helped them to overcome many of 
the barriers they faced initially in developing effective working relationships, as did their 
ability to visit the site of project services on a daily basis and to locate key staff onsite. 

The other two grantees, one a university and the other a state board for vocational 
education, were far removed from the location of project services. The University established 
relationships with three LEAs, two of which, located in distant states, implemented services in 
area vocational-technical centers, while the third (the one located nearest the university) 
established an alternative school. The state board developed relationships with 10 
independent school units (variously independent school districts, tribal organizations, and BIA 
schools) located at great distances from the state capital on or near four reservations across 
the state. None of these schools was a vocational school, which meant that the connections 
between the state board and the project sites was tenuous at best. Obtaining agreements from 
the schools and districts required a great deal of time and persuasion on the part of the project 
director, who worked in the state capital but spent much of her time traveling around the state 
on behalf of the project. While the impetus to establish a project under these conditions was 
laudable, the problems implicit in the distances and in the lack of any direct organizational 
control on the part of the grantee over the participating schools were nearly impossible to 
overcome. In combination with prevailing economic conditions and other factors, the paucity 



5-4 



ERIC 



131 



of positive participant outcomes in these sites is not suq)rising. At the same time, this 
project's experience is instructive in the context of attempts to develop and implement 
strategies for improving education in extreme rural areas of the country. 

Overall, then, our analyses suggest that a close linkage between the grantee and the 
location of project services is likely to facilitate both implementation and outcomes. While 
these close connections are not, by themselves, sufficient to ensure effective projects, their 
absence constitutes a barrier that is difficult to overcome in attempts to implement 
demonstration projects like those funded under CDP. 

Experience With the Model 

In general, our findings suggest that grantees interested in implementing strategies to 
improve the educational experiences and outcomes of at-risk youth achieve more early, and 
ongoing, success if they are implementing models they know well. The Sequoia Union High 
School District in California implemented the Academy model in two of the district's eight 
high schools. This district already had Academies in place in two other high schools, and the 
CDP grant's project director was responsible for implementation of the eariier Academies, 
which have been in place for about 10 years. Her familiarity with the model-and issues 
requiring resolution for its effective implementation-enabled the district to have most 
components in place, or at least arranged for, at grant award. Perhaps more important, her 
long career with the school district and high credibility with the implementing high schools' 
administrative and instructional staff facilitated effective response to ongoing problems that 
occurred over the course of the demonstration. In short, the high level of knowledge of this 
model, including its limitations as well as its advantages, facilitated its effective 
implementation and helps to explain the projects' success in terms of student outcomes. 

Similarly, staff at the Oklahoma Child Service Demonstration Center in Cushing had 
developed the service design they implemented at the area vocational-technical center. 
Originally intended for adolescents with learning disabilities, the model was adapted to 
address the needs of at-risk youth attending the vocational-technical center. Further, prior to 
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award, the grantee had already finalized plans for collaborating with the area center, which 
included a formal agreement with the center's administration to participate in the project and 
to follow the evaluation's intended design. These factors facilitated the project's successful 
implementation in fall 1989. Even so, the project's participant outcomes were modest, 
principally, we believe, because most of the project's services were available to all students 
(participants, control group members, and classmates in the vocational programs in which 
participants enrolled during the demonstration) and because the center required participants to 
meet the school's regular eligibility requirements, thus probably masking the effects of the 
intervention. 

Even when grantees planned to adapt or expand locally developed service designs, 
local conditions could interfere. The experience of the Detroit Public Schools is instructive in 
this regard. Funded to expand a model that the district initially developed for youth with 
limited proficiency in English, the project was unable to initiate services as planned because 
of major changes in the district. These changes included replacement of most of the school 
board, appointment of an interim superintendent whose charge was to reorganize the district, 
and a hiring freeze that precluded hiring a project director until well into the second semester. 
Nevertheless, even given the late start of this project, its strategy of providing supplementary 
support to students attending the city's area vocational-technical centers proved effective in 
terms of student retention, with students who received the special instructional support, 
counseling, and other project services significantly more likely to remain in school and to 
improve their grade point averages. 

Unlike these grantees, the two grantees that planned to replicate a nationally validated 
model. Project COFFEE, in a total of seven localities lacked experience with that model, a 
factor that probably exacerbated problems with its implementation. To facilitate local 
implementation, the Clemson project provided preimplementation training on COFFEE to 
representatives of each of the local sites that participated in the project, including a 
presentation by COFFEE'S originator and provision of materials on replicating COFFEE. 
Additionally, the project made extensive technical assistance available to the three sites, 
through visits, telephone calls, and an electronic network. However, even this level of 
assistance failed to overcome the various local administrative and other problems that two of 
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the sites faced in attempting to implement COFFEE in a way that would be acceptable to the 
administrators of the two area vocational schools that served as the location of project 
services. Lack of adequate funds to implement the model also exacerbated the effects of low 
administrative support these two sites faced as well, an inference supported by the fact that 
the third participating district, which was able to obtain additional funds, experienced greater 
success in replicating the model. 

It is important to point out that the problems faced by these and other sites attempting 
to replicate COFFEE were not necessarily the result of this model's complexity but rather at 
least in part the result of factors that had to be addressed during the projects' early stages. 
Also initially developed to serve youth identified for special education services (in this case 
youth with serious emotional disturbance), COFFEE is an alternative school model that differs 
more markedly from the organization of "regular" schools than many other dropout prevention 
initiatives. It requires a separate location, small class size, extra preparation and consultation 
time for teachers, an integrated academic/vocational curriculum, a student-operated business, 
availability of counseling, and other features, which make it expensive to implement. None 
of the funding levels proposed by, or awarded to, CDP grantees was sufficient to replicate 
this model, and decisions had to be made about which features to keep and which to drop in 
order to operate a project at all within the constraints of available funds. The changes that 
were necessary essentially diluted the model to the extent that intended outcomes were not 
achievable. 

Other problems also limited the replication of the COFFEE model. For example, 
another of the intended COFFEE replications, in Broward County, Florida, elected to establish 
a school-within-a-school in one of the district's area vocational-technical centers. Issues of 
administrative control over the project's decisions regarding selection of curriculum and 
instructional design as well as other aspects meant that the environment was not favorable to 
implementation of the project as intended by its local director, and participants' lack of 
success in outcomes measured by the evaluation reflects the problems experienced by the 
project. Additionally, the evaluation itself was less effectively implemented in that locality 
than elsewhere, based on our early misunderstanding of district policies and operational 
realities. For this reason, implementation may have been difficult in several respects. As 



noted in our discussion of the North Dakota experience, certain features of the model were 
incompaUble with state poHcies and regulations for education. Other features were not 
susceptible to implementation locally because of economic conditions, resource issues, and 
distance. It is possible that prior experience with the model might have ameliorated some, 
though certainly not all, of these problems. 

Only one site, Oconee County, was able to undertake a faithful replication of the 
model, using considerable additional funding obtained through a grant awarded by the state. 
In addition to following the essential features of the model, Oconee also targeted highly at- 
risk youth for services, which is essentially the population COFFEE intends to serve. 
Although Oconee students were not necessarily identified for special education services (a 
few were), they were youth with serious educational problems for whom the alternative 
school was probably a last resort, and project staff worked hard to implement a design that 
would enable them to reach these youth. 

Again, however, Oconee was the only site of the seven that intended to replicate 
COFFEE to experience much success in adopting the model. The others were in most 
instances providing important services to their target groups, but they were not replicating the 
model. Perhaps, then, one of the lessons from our evaluation is that this type of 
demonstration program may not be the appropriate context for attempts to replicate validated 
models unless the grantees have prior experience with the model they select and the resources 
needed to implement it fully. Rather, it may be more productive to support districts in 
expanding or adapting designs with which they have prior experience. The alternative is to 
arrange for sufficient technical assistance to ensure that localities develop the capacity to 
replicate some of the validated models. Additionally, decisionmakers responsible for 
allocating resources in demonstration programs need to be sensitive to the costs of these 
validated models. Absent sufficient resources, effective replication is not possible even given 
adequate technical assistance or other capacity-building activities. 
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Matching of Services to Targeted Youth 



Another factor that affects outcomes is the efficacy of particular service designs for 
participants. Project success hinges to a great extent on careful analysis of the types of youth 
targeted for services, and on provision of services that are appropriate to the specific target 
group. Outcomes achieved by the projects should be viewed in this light. It is clear that the 
definition of "at riskness" encompasses a broad range of youth, and a broad range of needs. 
For example, participants in Gushing' s GRADS and Sequoia's Academies were at risk, with 
educational and personal problems that had constrained their affiliation with and performance 
in school. However, these youth could be considered less seriously at risk than the groups 
targeted by some of the other projects. Gushing participants had to qualify for the vocational- 
technical center, which meant that they must be on track to graduate. Academy students had 
to demonstrate motivation, a certain reading achievement level, and be without severe 
behavioral problems. The soundness of these projects' designs for these students is suggested 
by their outcomes, as well as by the success of their implementation. 

On the other hand, even given effective implementation, committed staff, and a 
supportive environment, Oconee's efforts to improve the educational outcomes of a highly at- 
risk target population were likely to succeed slowly, if at all. Nevertheless, the project's early 
success was reflected in positive changes in some of the school affiliation outcomes, and to 
the extent that the project was able to encourage even some of these youth to finish school, it 
could be considered successful. Similarly, participants in the North Dakota projects 
experienced personal problems and environmental deprivations that would defeat most people. 
Even so, some of the project sites succeeded in encouraging participants to come to school 
more regularly, begin to think about their futures, and otherwise respond positively to their 
involvement in the project. 
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The Role of Academics 



Academic components of the projects ranged from pull-out tutoring to help students 
catch up in academic courses, which was one of the key components of the North Dakota 
projects, to the integrated academic-vocational curriculum of the Business Technology 
Academies. All of the projects, even those whose original design focused mainly on 
supplementary assistance in vocational courses, eventually paid some attention to participants' 
academic needs. Gushing* s GRADS project, for example, set up a learning center that 
contained materials intended to help students master their vocational course work. Early in 
the project's life, however, students began bringing their academic assignments to the center, 
and the two center staff devoted a fair amount of time to assisting participants with basic 
skills on the theory that such assistance would help students make the grades they needed to 
stay enrolled in vocational education as well as to perform better in their vocational courses. 

The Anne Arundel and Broward projects offered at least some academic course work 
at the area vocational-technical centers in which the projects operated. Anne Arundel 
implemented a computer-based vocational English course, for which participants received 
regular academic credit, that focused on communication skills and employability development. 
As part of its school-within-a-school design, the Broward project implemented a computer- 
assisted instruction system for participants' academic course work that the school's executive 
director selected. The project made tutoring assistance available to students who experienced 
difficulty. 

Finally, the two Academies and the Oconee alternative school provided academics as 
an integral component of their projects. All three projects implemented small classes, 
collaboration among teachers in planning courses and monitoring participants' progress, and 
integration of academic and vocational content. That this approach shows promise for 
improving academic performance and attitudes toward school is reflected in participants' 
outcomes. At one of the three, Carlmont Academy, participants became sufficiently engaged 
in their required math courses (algebra) that they successfully lobbied the project to add an 
Academy geometry course to the project. 
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The experience of the dropout prevention projects point to the importance of attention 
to both vocational and academic aspects of students' high school programs. In the past, 
many--perhaps most-high school vocational programs have been physically and 
psychologically dissociated from academics, in part because of the stereotypes vocational 
education has often suffered and in part because of physical separation of these two aspects of 
high school. Our study suggests that this separation is both unrealistic and misguided. Even 
in the projects that planned to provide supplementary assistance to participants' vocational 
work, staff inevitably found themselves attending to students' academic needs as well, 
through tutoring, coordinating with academic teachers, and teaching e-jademics in order to 
teach vocational content. The more comprehensive projects typically recognized the 
inseparability of the two components of high school in their efforts to integrate the two, as 
well as to avoid perpetuating the traditional negative stereotypes that vocational education has 
typically faced. Our observations of studsnts in the computer labs during lunch period, 
working on English papers, spreadsheets, and other vocational assignments supports the 
notion that a strategy that integrates academics into vocational curricula may help to engage 
students and improve their overall performance. 



The Importance of Personal Supports 

As noted earlier, the projects funded under CDP intended to affect students who had 
often experienced years of failure and frustration in school. Recognizing the difficulty of 
turning such youth around, many of the projects planned activities that would offer 
participants some level of adult supports beyond that available in the classroom. These 
included, for example, biweekly group counseling led by a licensed clinical psychologist; 
employment of specialized staff such as guidance counselors, instructional support staff, or 
attendance monitors; and implementation of mentoring. Perhaps equally important, virtually 
all of the projects in practice provided personal support to participants. Academic and 
vocational specialists in the North Dakota sites typically devoted much of their time with 
students to addressing personal concerns. The Oconee project director spent many hours 
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working with students, families, law enforcement, and others in attempts to help students 
resolve their serious personal problems. All of the projects located in area vocational- 
technical centers employed staff who provided student support, in addition to staff in charge 
of formal counseling activities. Perhaps least often acknowledged but equally important, 
many of the shops and labs that students spend much of their time in were organized in a 
way that provided a supportive environment. Typically, vocational classes are small; students 
often work in small groups or teams, collaborating on projects, and the instructor often 
functions more nearly as a "supervisor" than as a more traditional teacher. For many 
students, this environment, particularly in combination with other project components, 
appeared to facilitate engagement in a way that had not occurred in the more traditional 
school environment. Most of the projects with components that addressed participants- 
personal support needs showed improvements in students' affiliation with school; although 
our data do not permit us to say with confidence that these services by themselves made a 
difference, our anecdotal evidence suggests their importance in the overall success achieved 
by the projects. 
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The Efficacy of Vocational Education as a Dropout Prevention Strategy 

As noted in Chapter 3, at the secondary level the term "vocational education" covers a 
broad range of activities, from a class in career exploration to preparation for entry-level jobs 
in a specific occupation (such as motorcycle mechanics or cosmetology). By design and in 
some instances by default, the projects that participated in the evaluation varied widely in the 
types and amounts of "vocational education" they provided participants, and the efficacy of 
this service as a dropout prevention strategy must be viewed in this light. 

The projects located in area vocational-technical schools (Gushing, Detroit, Anne 
Arundel, Broward) provided services within the context of the vocational programs students 
were pursuing. For example, the Gushing project worked with vocational instructors to 
identify student needs, reviewed instructional materials, and developed more user-friendly 
materials based on analysis of reading levels and other factors. Additionally, the project 
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operated a learning center where students could obtain extra help with their vocational courses 
and work on computer programs keyed to vocational content. However, because most of 
these services were made available to all students, both project participants and control group 
members, their efficacy as dropout prevention was difficult to isolate.' While the project did 
achieve reduction in dropping out, outcomes in other areas might have been clearer had the 
"treatment" not been available to all students. At the same time, the efficacy of these 
"supplementary" services may not be sufficient to qualify as a true vocational intervention and 
thus may not be a fair test of vocational education in dropout prevention. 

The other area school projects also contained such "supplemental" services-provision 
of instructional ^upport staff in shops and labs, learning. center services, vocational and 
personal counseling. Positive outcomes accompanied these supports in Detroit (reduced 
dropouts, higher GPAs) and in Anne Arundel. In addition to these supplementary services, 
the Anne Arundel project implemented a computer-based, credit-bearing vocational English 
class and credit-bearing, paid work experience. The latter was intended to be related to 
students' vocational programs, but in general students were unsuccessful in finding jobs in 
their vocational areas and were permitted to take any jobs available as part of the work 
experience component. The combination of these components did yield improvements in 
students' GPAs as well as some improvement in school affiliation measures (perceptions of 
job preparation and view of the safety of the school environment). 

Overall, however, most of these projects did not really establish vocational education 
as an intervention, since most participants had already arranged to enroll in vocational 
education and received project services to supplement their vocational courses. This was not 
the case in Detroit, which recruited students to attend area schools in order to participate in 
the project, which may help to explain improvements in grades and persistence. A fairer test 
of vocational programs as a dropout prevention strategy might be examination of the extent t 
which selection of a vocational program in an area vocational-technical school instead of an 



'Further, since the school declined to adjust admission requirements for the demonstration project, students served 
may have been less at risk and thus more nearly comparable to control group members, thus reducing the potential 
of the treatment to show large differences. 
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academic program in a "regular" high school affects youths' likelihood to remain in school. 
This comparison was not really available to the evaluation. 

Our findings suggest that the type of vocational education provided to less severely at- 
risk youth through the Academy model does seem to work for those youth (given the caveat 
that the evaluation was unable to implement random assignment in the two Academy sites). 
The chief "vocational" component received by study participants in the two Academies was 
an integrated academic-vocational curriculum whose vocational emphasis was business 
technology provided in newly outfitted computer labs that housed up-to-date technology (part 
of which was donated by local businesses). Students worked on academic tasks (e.g., English 
papers) in the labs, and teachers worked together to plan assignments that would integrate 
their business technology with their academic course work. The evaluation's findings suggest 
that this curriculum succeeded in engaging participants, as evidenced by improvements in 
academic performance and school affiliation. It is interesting that the other "vocational" 
aspects of the traditional Academy model-mentors and paid work experience-did not work 
well in these two sites. Possibly because of economic conditions, jobs in fields related to 
students' course work were difficult to find. Further, many students already had jobs that, 
although they were not related to course work, paid better than related jobs; predictably, 
students were unwilling to give those jobs up. At least one of the Academies experienced 
difficulties in maintaining a mentoring program, with students apparently not much interested 
in having mentors and teachers not entirely successful in locating businessmen who would 
match up well with Academy students in terms of interests and personal characteristics. 
Further, given the existence of several other Academies in the area, it is possible that a 
saturation point may have been reached in terms of numbers of people who would be 
interested in making the commitment that this type of volunteer work requires. 

The Oconee alternative school represents another approach to vocational education. 
While the intent had been that students in their second year of enrollment at OASIS would go 
to the area vocational-technical school and enroll in a vocational program, reluctance on the 
part of vocational school administrators along with district policies regarding eligibility for 
enrollment at the area school made this option infeasible for most participants. As a 
COFFEE replication, however, the project did establish an entrepreneurial business- 
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manufacture and sale of picnic furniture--that engaged the interest of many students. The 
business offered youth the opportunity to learn many aspects of small business, including 
marketing and sales, accounting, ordering and production, etc. Given the rural nature of the 
project's location, the opportunity to learn multiple aspects of small business or self- 
employment could be considered particularly useful to individuals who would likely enter 
such employment following graduation. Its availability may help to explain the improved 
school affiliation experienced by project participants as well as the modest improvement in 
school performance. 

Finally, in part based on the EBCE model, the Portland project implemented a 
vocational mentoring model based on a formal agreement with a local hospital. The model 
involved students' exploration of health careers based on "shadowing" hospital employees, 
work with employees who served as mentors, and work experience in the hospital as well as 
conduct of some of their academic classes at the hospital. This model, intended in part to 
interest participants in pursuing health careers at the postsecondary level, yielded positive 
outcomes in terms of school affiliation and school performance, though it did not eventuate in 
reduced dropping out, based on the performance of a matched comparison group. 

One of the continuing claims of vocational educators is that youth who are alienated 
from the traditional "seat-based" academic model of high school education are more likely to 
remain in and complete high school if they have the opportunity to enroll in vocational 
education. At the same time, these educators generally agree that haphazard enrollment in 
vocational education that does not include a coherent, sequenced program leading to entry- 
level employment or further training at the postsecondary level may not offer much payoff to 
youth in terms of posthigh-school options. The extent to which this evaluation sheds light on 
these issues is mixed. On the one hand, some of the "newer" generations of activities that 
fall under the rubric "vocational education," such as the integrated academic-vocational 
curriculum of the Academy model, Portland's vocational mentoring (which is one component 
of a districtwide continuum of vocational options geared to the full range of student 
capabilities and interests), and the COFFEE model of student-operated business, appeared to 
engage students and may have facilitated positive outcomes, although not a demonstrated 
reduction in dropping out. 
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On the other hand, the demonstration program was not structured to test whether 
enrollment in a program of vocational education, in an area school, retained students, 
primarily because the projects that operated in these schools provided supplements to the 
basic program rather than the program itself. Further, in most localities, the CDP project's 
existence caused discomfort within the environment of those schools, with administrators 
concerned that at-risk students would disrupt the school environment and instructors 
concerned that project services might interfere with the amount of time available for teaching. 
The answer to the question of whether vocational programs retain potential dropouts will 
require a study that involves comparison of individuals who either do or do not enroll in such 
programs rather than comparison of those who receive supplemental services with those in the 
same programs who do not receive the extra services. The one project, Detroit, that 
apparently did recruit students to vocational education and compared those students with a 
control group not enrolled in vocational education did achieve reduced dropout rates. While 
given various implementation problems this finding should be viewed as tentative, its 
existence does suggest the advisability of further study under better conr -oiled conditions to 
provide an answer to the question of whether vocational education will both keep at-risk 
youth in school and improve their postschool economic prospects. 



Structural Changes 

Within the constraints of the high school environment as it currently exists, including, 
for example, increasingly complicated and time-consuming graduation requirements, limited 
resources to support "special" services for students who are headed toward failure and early 
departure, and increasingly rigorous responses to discipline and safety problems, the projects 
that participated in the evaluation experienced some success in designing and implementing 
strategies to improve the school experiences and outcomes for the at-risk adolescents they 
targeted for services. Two of the project models aimed to change all or most of the school 
experience. The school-within-a-school creates a smaller, more coherent environment for 
students on the theory that this environment will facilitate development of self-esteem and 
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motivation while not closing participants out completely from the activities of the larger 
school environment (e.g., extracurricular activities) for at least part of the day. These projects 
appear to succeed to the extent that they become identified as "special" in a positive way and 
that they develop the right mix of adults who have the skills and commitment to make the 
experience work for students. While the two schools that implemented this model did not 
achieve reduced dropping out, they did demonstrate a substantial number of school 
performance and affiliation improvements. 

The alternative school also has the advantage of changing the school experience of 
students who have failed in the regular environment by moving them to another place and 
then attempting to make that place work for them. To increase the likelihood of success, this 
type of intervention typically places strong emphasis on recruiting teachers and counselors 
with particular sensitivity toward and sympathy for highly at-risk youth, creating a structured 
and nurturing environment characterized by a high level of consistency in treatment of all 
students, and provision of substantial support as students begin to work through personal 
issues that often exacerbate their problems with school. This type of school is most likely to 
succeed with its target population to the extent that staff are able to create a community in 
which individuals begin to succeed in their work and hence to build the confidence and self- 
esteem that are key to success. The danger of the alternative school is the stigma that many 
students experience as being "different," or "dumb," or "failures." Even so, as the director of 
OASIS commented, some highly at-risk youth are willing to endure these stereotypes in order 
to find an environment in which they can begin to experience success, develop affiliation with 
their peers and teachers, and eventually, perhaps, persist to graduation. It is unfortunate that 
changes in key staff at the school occurred before the treatment really had much of a chance 
to "take" with participants. Even so, the modest improvements that the project's first cohort 
experienced suggests that this model is worth considering for districts seeking strategies to 
improve the outcomes of highly at-risk youth. 

Finally, projects that address only a part of students' school day can also achieve 
positive effects to the extent that they carefully design services to address the particular needs 
of targeted students. These projects typically used CDP funding to "add on" to the standard 
vocational programs available to youth attending area schools those supports thought to 
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facilitate success, including instructional support either in class or in a learning lab, 
development of instructional aids (including software and paper materials), and counseling to 
assist students in learning to function more effectively within their environments. While 
effects that such programs yield may be slower or smaller, the activities of projects like 
GRADS, STARS, and YES help to support students' vocational programs and, if well 
implemented, can be expected to improve not only educational outcomes but also posischool 
employment success. While the gains in student outcomes achieved by these projects were 
generally modest and in some cases nonexistent, given the reality that most participants had 
experienced difficulty in school for upwards of 10 years, even those outcomes suggest the 
legitimacy of ongoing initiatives that attempt to improve the educational experiences and 
outcomes of this segment of the nation's youth that may lead to improvements in their 
postschool employment and other options. 



Components of Ef fective Dropout Prevention Programs in Vocational Education 

Review of the experiences of the projects funded under the CDP demonstration from a 

summative perspective permits reflection on those components of the projects, taken together, 

that appear to have promise for improving the persistence and educational success of at-risk 

youth. Among the components that appear critical are the following: 

A smaller, more personal environment, such as that available in the school- 
within-a-school and alternative school environments; 

Vocational education, preferably that contains integration of academics with the 
vocational content, and, for most participants, has an occupational concentration 
leading to good entry-level jobs or continued training at the postsecondary 
level; 

A formal counseling component that incorporates attention to personal issues 
along with career counseling, employability development, and life skills 
instruction; 

• Formal, ongoing coordination of the academic and vocational components of 
participants' high school programs; 
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A structured environment that includes clear and equitably enforced behavioral 
expectations; and 

Personal, supportive attention from adults, through a mentoring or other project 
component. 
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2 


3 


2 


1 


16 


11 


1 


3 


6 


3 


1 


1 


27 


34 


37 


76 


27 


30 


30 


73 


Black 


23 


It 


26 


21 


1 


1 


3 


3 


11 


10 


32 


32 


0 


0 


0 


0 


0 


0 


0 


0 


White 


22 


42 


34 


34 


77 


74 


33 


28 


22 


28 


15 


24 


0 


0 


1 


3 


1 


1 


3 


5 


Hiapanic 


27 


25 


22 


30 


1 


I 


0 


0 


0 


0 


1 


3 


0 


0 


0 


0 


0 


0 


0 


0 


Asian/Pacific blander 


2 


8 


5 


9 


1 


1 


0 


0 


0 


0 


0 


4 


0 


0 


0 


0 


0 


0 


0 


0 


1 Afe AppropialB 


1 


54 


54 


53 


5t 


5t 


12 


7 


13 


13 


27 


35 


4 


7 


8 


26 


8 


9 


10 


35 


Uder 




35 


35 


42 


3S 


37 


26 


27 


26 


28 


22 


28 


23 


27 


30 


53 


20 


21 


23 


43 
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Introduction 



For each of the 12 sites that participated in the national evaluation, we have included a 
set of tables that report participant outcomes. The sites appear in the following order: 
Woodside Business Technology Academy, Woodside, California 
Carlmont Business Technology Academy, Carhnont, Califoniia 
Guidance in Retraining Adolescent Dropouts (GRADS), Gushing, Oklahoma 
OASIS Altemative School, Oconee County, South Carokina 
Youth Experiencing Success (Y.E.S.), Anne Arundel County, Maryland 
Changing How Our Pupils Succeed (C.H.O.P.S.), Broward County, Florida 
Vocational Continuum Program, Portland, Oregon 
Preparing At-Risk Youth for Employment, Detroit, Michigan 
Project COFFEE Vocational Training Program, Turtle Mountain, North Dakota 
Project COFFEE Vocational Training Program, Fort Totten, North Dakota 
Project COFFEE Vocational Training Program, Fort Berthold, North Dakota 
Project COFFEE Vocational Training Program, Fort Yates, North Dakota 
For each site, the appendix contains six tables, as foUows: 
Table 1: Characterstics of Study Participants 

Table 2: Analysis of School Performance Outcomes, by Cohort by Year 

Table 3: Gap Reduction Analysis of School Performance Outcomes, by Cohort 
by Year 

Table 4: Analysis of Project Dropout Rates, by Cohort by Year 

Table 5: Analysis of School Affiliation and Student Perception Outcomes, by 
Cohort by Year 

Table 6: Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
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Table B.1.1 



Characteristics of Study Participants 
Woodside Business Technology Academy 











1 




Gfoop 




Gr«ap 






muj.Mjj.uAujjmu.LiMjjj.uj_w^^ 








«i 


% 








% 




ft 




% 






Female 


18 


44% 


24 


49% 


21 


60% 


16 


40% 


39 


51% 


40 


45% 


Male 


23 


56% 


25 


51% 


14 


40% 


24 


60% 


37 


49% 


49 , 


55% 


RacetEOatkiti Of-yS 




Native American 


0 


0% 


2 


4% 


2 


6% 


1 


3% 


2 


3% 


3 


3% 


Black 


13 


32% 


4 


8% 


10 


29% 


7 


18% 


23 


30% 


11 


12% 


White 


12 


29% 


24 


49% 


10 


29% 


18 


45% 


22 


29% 


42 


47% 


Hispanic 


15 


37% 


16 


33% 


12 


34% 


9 


23% 


27 


36% 


25 


28% 


Asian/Pacific Islander 


1 


2% 


3 


6% 


1 


3% 


5 


13% 


2 


3% 


8 


9% 




Age appropriate 


27 


66% 


30 


61% 


22 


63% 


24 


60% 


49 


65% 


54 


61% 


Overage for grade 


14 


34% 


19 


39% 


13 


37% 


16 


40% 


27 


36% 


35 


39% 



♦Sample sizes for individual characteristics vary slightly across characteristics and cohorts. 



Table B-1.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Woodside Business Technology Academy 



iiiilii^SiSi 
OuUsotne 


Coltorll 


Cohort t 


Combined 




1990.91 


1991-92 


1990*91 


1991*92 


Year 1 




Dropout Status 
















Higher GPA 






-G,-A 


GA^ 




G,A^R 




Higher number 
of credits 
eamed 


AJl 






G,A^ 




G,A^ 




Fewer absences 










-G 




-G 


Fewer courses 
failed 






-G 


G,A 




G,A 





Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 
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Table B-1.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Woodside Business Technology Academy 





Cohort 1 


Cofeott t 


Cotnhined 
Cohort 












Year 1 


Y€»r2 


Higher GPA 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


Higher number 
of credits 
earned 
















Fewer absences 
















Fewer courses 
failed 


+ 


+ 


+ 


+ 


+ 




+ 



Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-1.4 



Analysis of Project Dropout Rates, by Cohort by Year 
Woodside Business Technology Academy 



Year 


Coliortl 






Ytar 






Tr«afnaent 
Group 

(Na40) 


Cootrol ; 
Group 


Tr«ati»ent 
Group 


Control 
Group 


Trea 
Gr 


tmejtt 
oup 


CJo 
Gi 
(N 


utrol 
*oup 


n 


% ■■■ 


,:-n 


. . 


■■ n ■: 






% 


a 




« 


. % 


1989-90 


1 


3% 


0 . 


0% 










Year 1 


3 


4% 


2 


2% 


1990-91 


3 


8% 


4 


9% 


2 


6% 


2 


5% 


Year 2 


5 


7% 


7 


8% 


1991-92 


4 


10% 


5 


11% 


2 


6% 


3 


8% 







Table 8-1.5 

Analysis of School Afniiation and Student Perception Outcomes, by Cohort by Year 

Woodside Business Technology Academy 





;::|:$S|i::>!*::^^;::|:S:f^^ 








Combined 
Cohort 






, 'p»i^'' f'y$^n 






StkiHaAjBfaktum 














Schocd thoQ^t safi» 


AJfl 










G 




Teaching^eacheis better 












na 


na 


Better job preparation 
















Counseling/Counselors better 






A 






na 


na 


More academic 
encouragement 


















Classmates should not 
misbehave 














GAR 


Future expectations are better 






-R 




-G,-A 


na 


na 


Many classmates are 
college bound 






-A 


J 









Note: Statistical significance at p <05 is reprssented by G (gain score analysis), A (analysis of covariance), 
and R (reliability adjusted analysis of covariance). 
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Table B-1.6 



Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Woodside Business Technology Academy 





Cohort 1 


ColKHrtl 










im-n 










««»!8t»WBIKaBBB3»SSS 


--^ 


Schod thought safer 
















Teadiing/Teachera better 












na 


na 


Better job preparation 
















Counseling/Counselors better 












na 


na 


More academic 
encouragement 
























Gassmates should not 
misbehave 
















Future expectations are better 












na 


na 


Many classmates are 
college bound 

















Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-2.1 

Characteristics of Study Participants 
Carlmont Business Technology Academy 



Charactoistu:* 






CBbm2 


€onbia«d Coimi 


.; Group ■ ; 


Control 
Group 

(N^48) 




Cfintrcit 
Group 


Groap 








■■ ■ -'/-^ 








V|?::B;:;Si 




s 


% 


fk 


% 


Gender 


Female 


15 


37% 


22 


46% 


25 


52% 


16 


34% 


40 


45% 


38 


40% 


Male 


26 


63% 


26 


54% 


23 


48% 


31 


66% 


49 


58% 


57 


60% 


Race/EAnieky 




Native American 


0 


0% 


0 


0% 


2 


4% 


1 


2% 


2 


2% 


1 


1% 


Black 


12 


29% 


7 


15% 


14 


29% 


14 


30% 


26 


29% 


21 


22% 


White 


16 


39% 


15 


31% 


18 


38% 


19 


40% 


34 


38% 


34 


36% 


Hispanic 


10 


24% 


19 


40% 


12 


25% 


11 


28% 


22 


25% 


30 


32% 


Asian/Pacific Islander 


3 


7% 


7 


15% 


2 


4% 


2 


4% 


5 


6% 


9 


9% 


Age 


Age appropriate 


29 


71% 


29 


60% 


25 


52% 


24 


51% 


54 


61% 


53 


56% 


Overage for grade 


12 


29% 


19 


40% 


23 


48% 


23 


49% 


35 


39% 


42 


44% 



1 V ^- 'Sample sizes vary slightly across characteristics for each cohort 
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Table 8-2.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Carlmont Business Technology Academy 





Cohort I 




Cohort 2 




Outcome 








1990-91 


1991-92 


Vear 1 




Dropout Stams 






+ 








+ 


Higher GPA 




G 


G 


G 




G 


G 


Higher number 
of credits 
earned 












GA 




Fewer absences 














-G 


Fewer courses 
failed 


R 










R 


GAJE^ 



Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 
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Table B-2.3 



Gap Reduction Analysis of Scliool Performance Outcomes, by Cohort by Year 
Carimont Business Technology Academy 







Cohoit I 


Cohort I 












f 

1991*93 




Year 2 


Higher GPA 


+ 


+ 


+ 


+ 


+ 


+ 


+ 


Higher number 
of credits 
earned 
















Fewer absences 
















Fewer courses 
failed 


+ 




+ 


+ 


+ 




+ 



Note; Direction of statistic?' significance at p <.05 is represented by +/-. 
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Table 8-2.4 



Analysis Project Dropout Rates, by Cohort by Year 
Carbnont Business Technology Academy 



Year 


Cohort 1 






Ye«r 


Coatb&ted Coltort 


Treatment 
<H = 41) 


C<mtroi 


Treatmeiil; 


Cotttr<dl 




Co 

m 


nfrol 


n 




V.;.- 




n 






% 


liili 




illlllt 


% 


1989-90 


0 


0% 


3 


6% 


na 


aa 


aa 


na 


Year 1 


0 


0% 


6 


6% 


1990-91 


1 


2% 


8 


17% 


0 


0% 


3 


6% 


Year 2 


5 


6% 


15 


16% 


1991-92 


0 


0% 


9 


19% 


4 


9% 


7 


15% 







IV 



IVv 
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Table B-2.5 



Analysis of School Afniiation and Student Perception Outcomes, by Cohort by Year 

Carlmont Business Technology Academy 















1 

Mftt" ' 


, 1 . 








MMWII 


V1"lf""/ 




■ ■ i ^ ■•w^wwwuw*^ 








Scbod thoaght safer 












A 




Teaching^eachefs better 


AJR 






R 




na 


na 


Better job preparation 
















Counseling/Counselors better 












na 


na 


More academic racouragetnent 


R 














Smdkttt pttxtptkm 








Classmates should not 
misbehave 














GAR 


Future e:^pectatioDs are better 












na 


na 


Many classmates are 
college bound 

















Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analyst of covariance), 
and R (reliability adjusted analysis of covariance). Direction is presented by +/-. 
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Table B-2.6 



Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Carimont Business Technology Academy 



^^^^^^^^^^^^ ^ 






CMMrt 




m^n 1 tim-n 




im-n 


Ye«r t 















School thought safer 


+ 














Teaching/Teachers better 












na 


na 


Better job preparation 


+ 


+ 












Counseling/Counselors better 












na 


na 


More academic encouragement 
















- 




Qassmates should not 
misbehave 
















Future expectations are better 












na 


na 


Many classmates are 
college bound 


+ 















Note: Direction of statistical significance at p <05 is represented by +/-. 
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Table B-3.1 



Characteristics of Study Participants 
:ushing Guidance in Retraining Adolescent Dropouts (GRADS) 





Coboril 


C«3t«rt2 










Contrqi 
Grmp 

(N5;40> 


(Na47) 


Cwtrpt 
{1^*48) 




C«NfipiMin». 




11 


i % 


a 


% 


□infill' 


% 


tt 


% 


s 




« 




Gender 




Female 


11 


22% 


13 


33% 


17 


36% 


21 


44% 


28 


29% 


34 


39% 


Male 


38 


78% 


27 


68% 


30 


64% 


27 


56% 


68 


71% 


54 


61% 


Race/EthttieMy '::-^^' -\''Lr'--'-Z:i:'-^ 




Native American 


9 


18% 


6 


15% 


7 


15% 


5 


10% 


16 


17% 


11 


13% 


Black 


1 


2% 


0 


0% 


0 


0% 


1 


2% 


1 


1% 


1 


1% 


White 


38 


78% 


32 


80% 


39 


83% 


42 


88% 


77 


80% 


74 


84% 


Hispanic 


1 


2% 


1 


3% 


0 


0% 


0 


0% 


1 


1% 


1 


1% 


Asian/Pacific Islander 


0 


0% 


1 


3% 


1 


2% 


0 


0% 


1 


1% 


1 


1% 


Age 




Age appropriate 


33 


67% 


22 


55% 


25 


53% 


29 


60% 


58 


60% 


51 


58% 


Overage for grade 


16 


33% 


18 


45% 


22 


47% 


19 


40% 


38 


40% 


37 


42% 



♦Sample sizes for individual characteristics vary slightly across diaiacteiistics and coboits. 
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Table B-3.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Cmhing Guidance in Retraining Adolescent Dropouts (GRADS) 



Outcome 


Cohort i 


Cohort 2 


Cohort 


1989-90 


1990-91 


1991-92 


1990-91 




Year 1 


y«ir2 


Dropout Status 










+ 




+ 


Higher GPA 
















Higher number 
of credits 
earned 
















Fewer absences 
















Fewer courses 
failed 

















Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 



BEST COPY AVAILABLE 

ERIC B-15 



Table B-3.3 

Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Gushing Guidance in Retraining Adolescent Dropouts (GRADS) 



i^iffi'iiiiilMBil 




Cohort 1 


Cobon 2 






Outcome 




1990*91 


im.92 


1996*91 


I99i»92 


4i ' '"-^^ 
Year 1 


III... *..| 


Higher GPA 


+ 


+ 




+ 


+ 


+ 


+ 


Higher number 
of credits 
eamed 


+ 


+ 




+ 


+ 




+ 


Fewer absences 
















Fewer '"ourses 
failed 


+ 


+ 




+ 


+ 




+ 



Note: Direction of statistical significance at p <,05 is represented by +/-. 
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Table B-3.4 



Analysis of Project Dropout Rates, by Cohort by Year 
Gushing Guidance in Retraining Adolescent Dropouts (GRABS) 



I 



Year 


Cohort t 


C<iiMiri2 








Treatment 
Group 


Cotttrot 
<5ro«pf 
(N«39) 


Treatment 
Group 


CwtroJ 
Groop 




Co 

€^ 


wtrol 


ft 




s 




s 




« 




m 


' 


a 




1989-90 


4 


9% 


4 


10% 


na 


na 


na 


na 


Year 1 


6 


6% 


10 


11% 


1990-91 


7 


15% 


10 


26% 


2 


4% 


6 


13% 


Year 2 


9 


10% 


19 


22% 


1991-92 


10 


21% 


13 


33% 


2 


4% 


9 


19% 
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Table B-3.5 



Analysis of School Affiliation and Student Perception Outcomes, by Cohort by Year 
Cushing Guidance in Retraining Adolescent Dropouts (GRADS) 





Cohort 1 


C«fM>rt2 


COlMMl 














$ehaotA0U«tion 


School thought sa£sr 
















Teaching/Ieacbeis better 












na 


na 


Better job preparation 
















Counseling/Counselors bener 












na 


na 


More academic encouragement 


-A,-R 










q;r 






Classmates should not 
mijbchave 
















Future expectations are better 




A 






-A 


na 


. na 


Many classmates are 
college bound 

















Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis of covariance), and R 
(reliability adjusted analysis of covariance). 
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Table B-3.6 



Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Cushing Guidance in Retraining Adolescent Dropouts (GRADS) 























tm-n 




Yitrl j 




















School thou^t safer 
















TeadiingA'eacbeis better 












na 


na 


Better job preparation 










+ 




+ 


Counseling/Counselors better 












na 


na 


More academic encouragement 
























Classmates should not 
misbehave 
















Future expectations are better 












na 


na 


Many classmates are 
college bound 

















Note: Direction of statistical significance at p <.05 is represemed by +/-. 
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Table B-4.1 



Characteristics of Study Participants 
Oconee County Alternative School (OASIS) 



CO 

























wlii 




liiil 




mil 


a 


mil 


B 


% 




% 


Gender 


Female 


3 


31% 


12 


46% 


5 


42% 


2 


25% 


13 


34% 


14 


41% 


Male 


18 


69% 


14 


54% 


7 


58% 


6 


75% 


25 


66% 


20 


59% 


RacetESinieUy 


Native American 


1 


4% 


2 


8% 


0 


0% 


1 


mo 


1 


3% 


3 


9% 


Black 


1 


4% 


2 


8% 


2 


17% 


1 


13% 


3 


8% 


3 


9% 


White 


23 


92% 


22 


85% 


10 


83% 


6 


75% 


33 


89% 


28 


82% 


Hispanic 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Asi?a/Pacific Islander 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Age 


Age appropriate 


11 


42% 


7% 


27% 


1 


8% 


0 


0% 


12 


32% 


7 


21% 


Overage for grade 


15 


58% 


19 


73% 


11 


92% 


8 


100% 


26 


68% 


27 


79% 



18^ *Sample sizes for individual characteristics vary slightly across characteristics and cohorts. 
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Table B-4.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Oconee County Alternative School (OASIS) 



QUtCOttW 


Cohort 1 




Cohort 2 


Conibintid 
Cohort 


1989*90 


1990.91 


1991-92 


1990-91 


1991*92 


Year 1 




Dropout Status 
















Higher GPA 
















Higher number 
of credits 
earned 














G 


Fewer absences 
















Fewer courses 
failed 






G,A 











Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 
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Table B-4.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Oconee County Alternative School (OASIS) 





Cohort I 


Cohort i 


Comtiimd 




1990.91 


1991*92 


1990-91 


I991-9a 


Year 1 




Higher GPA 




+ 




* 


* 


na 


na 


Higher number 
of credits 
earned 








* 


* 


na 


na 


Fewer absences 






+ 


* 


* 


na 


na 


Fewer courses 
failed 








* 


* 


na 


na 



Note: Direction of statistical significance at p <.05 is represented by +/-. 
* Cohort without gap reduction group 
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Table 



Analysis of Project Dropout Rates, by Cohort by Year 
Oconee County AUemative School (OASIS) 



Ytar 


Coliortl 






Year 


C«n6ttte 


(i€^*ort'v 'r 1 


IVtatment 
Group 
(N=t25) 


Control 
<5ro«p 


Treatment 
<Jro«p 


Control 
Gro«p 


Trot 


tmetfi 


Coi 


' 1 
I 


tt 


% 


s 




s 


% 


« 


% 




tt 






% 1 


1989-90 


1 


4% 


3 


12% 


na 


na 


na 


na 


Year 1 


7 


20% 


8 


24% 1 


1990-91 


9 


36% 


10 


38% 


6 


60% 


5 


63% 


Year 2 


16 


46% 


15 


44% 


1991-92 


11 


44% 


15 


58% 


7 


70% 


5 


63% 
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Table B-4.5 



Analysis of School Affiliation and Student Perception Outcomes, by Cohort by Year 
Oconee County Alternative School (OASIS) 





■■■:.:--C<)hort 1" ':^^:yH!::^ry::;: 


Cohort;]: 




^;/p.::/Ptt*<».'ne ■ 


1989-90 


1990^91 


1991*92 


1990*91 








SchootA^iUation 


School thought safer 








A 








Teaching/Teachers better 


R 










na 


na 


Better job preparation 




A 












Counseling/Counselors better 












na 


na 


More academic encouragement 






A.R 










Student perceptions 


Classmates should not 
misbehave 














-G 


Future expectations are better 








A 




na 


. na 


Many classmates are 
college bound 

















Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis of covariance), and R 
(reliability adjusted analysis of covariance). 
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Table B-4.6 



Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Oconee County Alternative School (OASIS) 





















- 


School tbou{^ safer 




+ 


+ 


* 




na 


na 


TeacfaingA'eacbeis better 








* 




na 


na 


Better job preparation 


+ 


+ 


+ 


* 




na 


na 


Counseling/Counselors better 


+ 






* 




na 


na 


More academic encouragement 








* 




na 


na 


SmitHt ptmpHotts 


Classmates should not 
misbehave 










* 


na 


na 


Future expectations are better 










* 


na 


f— 

na 


Many classmates are 
college bound 












na 


na 



Note: Direction of statistical significance at p <.05 is represented by +/-. 
♦ Cohort without gap reduction group 
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Table B-5.1 

Characteristics of Study Participants 
Ann Arundel County YouA Experiencing Success (YJEJi.) 











< 










Tmtmmt 


it* ^23} 














% 


■■■a. 


% 










B 


% 


B. 




Ciendet 




Female 


4 


19% 


5 


22% 


5 


28% 


8 


44% 


9 


23% 


13 


32% 


Male 


17 


81% 


18 


78% 


13 


72% 


10 


56% 


30 


77% 


28 


68% 


Race/EAukitf 




Native American 


6 


29% 


3 


13% 


0 


0% 


0 


0% 


6 


15% 


3 


7% 


Black 


4 


19% 


3 


13% 


7 


39% 


7 


39% 


11 


28% 


10 


24% 


White 


11 


52% 


17 


74% 


11 


61% 


11 


61% 


22 


56% 


28 


68% 


Hispanic 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Asian/Pacific Islander 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Afi^ 






Age apprrpriate 


4 


19% 


6 


26% 


9 


50% 


7 


39% 


13 


33% 


13 


32% 


Overage for grade 


17 


81% 


17 


74% 


9 


50% 


11 


61% 


26 


67% 


28 


68% 



♦Sample sizes for individual charsicteristics vary slightly across cbaracteiistics and cohorts. 
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Table B-5.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Ann Arundel County YouA Experiencing Success (YM.S.) 





Cohort t 


CoboflZ 




0 t&Mttt 










1991-92 


Year 1 




Dropout Status 
















Higher GPA 










G 




0 


Higher number 
of credits 
earned 
















Fewer absences 


R 














Fewer courses 
faUed 






-R 











Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 
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Table B-5.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Ann Arundel County Youth Experiencing Success (YJS.S.) 







Coltoit t 








W91-92 


1990-91 


1991.92 


Vear 1 




Higher GPA 


+ 


+ 


+ 








+ 


Higher number 
of credits 
earned 
















Fewer absences 
















Fewer courses 
failed 






+ 











Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-5.4 



Analysis of Project Dropout Rates, by Cohort by Year 
Ann Arundel County Youth Experiencing Success (YJES.) 



to 



Year 


Cohorf 1 






Year 






Treafineat 
Group 




Treatment 
Group 
<N= 18) 


Control 
Group 


Trca 
Gr 


It? 


C« 
Gi 
(N 


-41) 


n 


■'■ %;M;, 






n ■ 




tt 


% 


8 




iillii 


■ % 


1989-90 


0 


0% 


2 


9% 


na 


na 


na 


na 


Year 1 


4 


11% 


5 


12% 


1990-91 


9 


47% 


10 


43% 


4 


22% 


3 


17% 


Year 2 


13 


35% 


14 


34% 


1991-92 


9 


47% 


11 


48% 


4 


22% 


4 


22% 
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Table B-5.5 



Analysis of School AfTiliation and Student Perception Outcomes, by Cohort by Year 
Ann Arundel County Youth Experiencing Success (Y.E.S.) 



liililiiiSiiiiiiliiilHIiiSl 


C«hart 1 


Cohort 2 




1989*90 












¥«ur2 


SehooSAffltUiUm 
















School thought safer 










A 




A 


TeachingA"eachers better 












na 


na 


Better job pieparatioQ 




A 












Counseling/Counselors better 












na 


na 


More academic encouragement 






















Qassmates should not 
misbehave 
















Future expectations aie better 












na 




Many classmates are 
college bound 

















Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis of covariance). and R 
(reliability adjusted analysis of covariance). 
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Table B-5.6 



Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Ann Arundel County Youth Experiencing Success (Y.E.S.) 













fii*D,hiiMfii'i 








sm-n 


















School tixMight safer 
















Teacfaiog^eacbeis better 












na 


na 


Better job preparation 
















Counseling/Counselors better 












na 


na 


More academic encouragement 
















^vitHt-tHttt^artiim 






Classmates should, not 
misbehave 
















Future expectations are better 












na 


na 


Many classmates are 
college bouixl 








+ 









Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-6.1 

Characteristics of Study Participants 
Broward County Changing How Our Pupils Succeed (CJB.OJPS.) 







CoiMMrtl 




IBliiilii 






C«ntr«A 


<Jroop 

{N* ) 






illll 




% 








% 


liiiiliH 




n 




Gen4er ' -^^ 


Female 


oa 




na 


na 


4 


15% 


5 


17% 


na 


na 


na 


na 


Male 


na 


na 


na 


na 


22 


85% 


25 


83% 


na 


na 


na 


na 


RaceiEAnkitf 


Native American 


na 


na 


na 


na 


6 


25% 


2 


7% 


na 


na 


na 


na 


Black 


na 


na 


na 


na 


5 


21% 


5 


18% 


na 


na 


na 


na 


White 


na 


na 


na 


na 


11 


46% 


16 


57% 


na 


na 


na 


na 


Hispanic 


na 


na 


na 


na 


2 


8% 


5 


18% 


na 


na 


na 


na 


Asian/Pacific Islander 


na 


na 


na 


na 


0 


0% 


0 


0% 


na 


na 


na 


na 




Age appropriate 


na 


na 


na 


na 


3 


13% 


1 


5% 


na 


na 


na 


na 


Overage for grade 


na 


na 


na 


na 


21 


88% 


21 


95% 


na 


na 


na 


na 



♦Sample sizes for individual characteristics vary slightly across characteristics and cohorts. 



Table B-6.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Broward County Changing How Our PupUs Succeed (CJH.O.P.S.) 





Coliort 1 


Cohort t 








1991-92 




im-n 


Vear 1 




Dropout Status 


na 


na 


na 






na 


na 


Higher GPA 


na 


na 


na 






na 


na 


Higher number 
of credits 
earned 


na 


na 


na 






na 


na 


Fewer absences 


na 


na 


na 






na 


na 


Fewer coxirses 
failed 


na 


na 


na 






na 


na 



Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantci-Haenszel test. 
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Table B-6.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Broward County Changing How Our PupUa Succeed (CJ[,OJ*.S.) 





Cohort 1 


Cohort Z 


1 Comltt^cKt 

Cohort 


1989*90 


im-n 








Year 1 




Higher GPA 


na 


na 


na 






na 


na 


Higher number 
of credits 
earned 


na 


na 


na 






na 


na 


Fewer absences 


na 


na 


na 






na 


na 


Fewer courses 
failed 


na 


na 


na 






na 


na 



Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-6.4 



Analysis of Project Dropout Rates, by Cohort by Year 
Broward County Changing How Our Pupils Succeed (CJB.OJPS.) 



Htm: 






IVeatmenf 

0i» ) 


Control 

€Pr« > 


Treatmeaf 

^^• « 19) 


CiHktrdE 


tt 


% 






ft 




n 


% 


1989-90 


na 


na 


na 


na 


na 


na 


na 


1 

na 


1990-91 


na 


na 


na 


na 


3 


16% 


4 


25% 


1991-92 


na 


na 


na 


na 


6 


32% 


7 


44% 



Table B-6.5 

Analysis of School Affiliation and Student Perception Outcomes, by Cohort by Year 
Broward County Changing How Our Pupils Succeed (C.H.O.P.S.) 





Cohort 1 


Cohort 1 








1991.91 














■ff^CSt---\i 


School thought safer 


na 


na 


na 






na 


na 


TeacfaingA'eacbeis better 


na 


na 


na 






na 


na 


Better job preparation 


na 


na 


na 






na 


na 


Counseling/Counselors better 


na 


na 


na 






na 


na 


More academic encouragement 


na 


na 


na 






na 


na 


Student pttctpHoit* ~ ■ i'<- 


Classmates should not 
misbehave 


na 


na 


na 






na 


na 


Future e}q>ectations are better 


na 


na 


na 






na 


. tui 


Many classmates are 
college bound 


na 


na 


na 






na 


na 



Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis of covariance), and R 
(reliability adjusted analysis of covariance). 
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Table B-6.6 



Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Broward County Changing How Our Pupils Succeed (C.H.O.P.S.) 



















tm-n 




1991-92 


Yuri 










School thought safer 


na 


na 


na 






na 


na 


Teaching/Teachers better 


na 


na 


tia 






na 


na 


Better job preparation 


na 


na 


na 






na 


na 


Counseling/Counselors better 


na 


na 


na 






na 


na 


More academic encouragement 


na 


na 


na 






na 


na 


StudtHt DtretpHom 






Classmates should not 
misbehave 


na 


na 


na 






na 


na 


Future expectations are better 


na 


na 


na 






na 


na 


Many classmates are 
college bound 


na 


na 


na 






na 


na 



Note: Direction of statistical significance at p ^.05 is represented by -i-/-. 
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Table B-7.1 



Characteristics of Study Participants 
Portland Vocational Continuum Program 



i 










. ^iiiij iii^Miinimi iiiiiniiiiiijMijinMii,Miiiii iiiii 

' ' rriiTtiirtrrtiiiit 

111 p II » PI * ' ^^T'S'S'T:- 






ISrtiHagat. 
















% 


11 


* 


s 


% 


« 






i 




% 


Oankr 




Female 


5 


22% 


14 


40% 


14 


54% 


15 


52% 


19 


39% 


29 


45% 


Male 


18 


78% 


21 


60% 


12 


46% 


14 


48% 


30 


61% 


35 


55% 






Native American 


0 


0% 


1 


3% 


1 


4% 


0 


0% 


1 


2% 


1 


2% 


Black 


20 


87% 


14 


40% 


12 


46% 


18 


62% 


32 


65% 


32 


50% 


White 


2 


9% 


16 


46% 


13 


50% 


8 


28% 


15 


31% 


24 


38% 


Hispanic 


1 


4% 


2 


6% 


0 


0% 


1 


3% 


1 


2% 


3 


5% 


Asian/Pacific Islander 


0 


0% 


2 


6% 


0 


0% 


2 


7% 


0 


0% 


4 


6% 




Age appropriate 


11 


48% 


16 


47% 


16 


62% 


19 


66% 


27 


55% 


35 


56% 


Overage for grade 




52% 


18 


53% 


10 


38% 


10 


34% 


22 


45% 


28 


44% 



'Sample sizes for individual characteristics vary slightly across characteristics and cohorts. 
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Table B-7.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Portland Vocational Continuum Program 





Cohort I 


Cohort t 


Coh 






1991-92 


1990-91 1 1991*92 


Year 1 




















Higher GPA 














G 


Higher number 
of credits 
earned 


-A 




G 




G,A 


-G 


G 


Fewer absences 








-G 


A^ 




A^ 


Fewer courses 
failed 




G 










G 



Note- Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 



BEST COPY AVAILABLE 



ERiC B-39 



Table B-7.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Portland Vocational Continuum Program 









Cohott 2 


Combine^i 
Cohort 










1990.91 


1991-.92 






Higher GPA 










+ 






Higher number 
of credits 
earned 






+ 




+ 






Fewer absences 
















Fewer courses 
failed 




+ 


+ 






+ 


+ 



Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table 8-7.4 



Analysis of Project Dropout Rates, by Cohort by Year 
Portland Vocational Continuum Program 









Year;;- 










Cmnb&M 




(N«23) 


Control 
Group 

{N«24) 


Treatment 
Group 


Control 
Crroitp 


Year 


Gr 


»44> 


Co) 
G> 
<N 


oirci 
onp „ 


n : 




» 




tt 


% 


n 1 % 




% 


tt 


% 


1989-90 


1 


4% 


3 


13% 


na 


na 


na 


na 


Year 1 


6 


17% 


10 


19% 


1990-91 


5 


22% 


5 


21% 


5 


24% 


7 


24% 


Year 2 


10 


23% 


14 


26% 


1991-92 


7 


30% 


4 


17% 


5 


24% 


9 


31% 


I 





2 1 7 



2 it) 
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Table B-7.5 

Analysis of School AfTiliation and Student Perception Outcomes, by Cohort by Year 

Portland Vocational Contihuum Program 











Coltert 


Com 


i>illMNi 


1989^ 




mun 








1 — 






■ 


School thought safer 




R 












Teacfaing/Teachers better 












na 


na 


Better job preparation 












GAR 




Counseling/Counselors better 


R 










na 


na 


More academic eacouiagement 






A 




-G 












Classmates should not 
misbehave 
















Rituie expectations are better 








A 




na 


. na 


Many classmates are 
college bound 

















Statistical significance at p <05 is represented by G (gain score analysis), A (analysis of covariance), and R 
(reliability adjusted analysis of covariance). 



ERIC 



CIO 



BA2 



Table B-7.6 



Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Portland Vocational Continuum Program 



Outcome 








mt 






mt-n 


tm-n 


mt-n 






School thought safer 










+ 




+ 


TeacfaingA'eacbeis better 












na 


na 


Better job preparation 
















Counseling/Counselors better 












na 


na 


More academic encouragement 




















Classmates should not 
misbehave 
















Future cjqwctations arc better 












na 


na 


Many classmates are 
college bound 




+ 













Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-8.1 

Characteristics of Study Participants 
Detroit Preparing At-Risk Youth for Employment 





Co 


hottl 






Charactertsdc* 


Tnm 
«r 
(N 


tmmt 
oap 

- ) 


(Ns: ) 


Treatment 


Omtttit 


Treatnmit 
Grottp 

{Nft ) 


Compwriaoa 
Gnmp 
<N= ) 




a 


% 


■■ n .:■ 


% 




% 


A 


% 


n 


* 


B 




Gender 








Female 


na 


na 


na 


na 


26 


30% 


30 


30% 


na 


na 


na 


na 


Male 


na 


na 


na 


na 


61 


70% 


69 


70% 


na 


na 


na 


na 


Race/Ethnieiey 












Native American 


na 


na 


na 


na 


0 


0% 


3 


3% 


na 


na 


na 


na 


Black 


na 


na 


na 


na 


76 


87% 


77 


78% 


na 


na 


na 


na 


White 


na 


na 


na 


na 


4 


5% 


8 


8% 


na 


na 


na 


na 


Hispanic 


na 


na 


na 


na 


6 


7% 


11 


11% 


na 


na 


na 


na 


Asian/Pacific Islander 


na 


na 


na 


na 


1 


1% 


0 


0% 


na 


na 


na 


na 


Age 








Age appropriate 


na 


na 


na 


na 


18 


21% 


34 


34% 


na 


na 


na 


na 


Overage for grade 


na 


na 


na 


na 


69 


79% 


65 


66% 


na 


na 


na 


na 



♦Sampk sizes for individual cbaracteristics vary slightly across chararteristics and cohorts. 



Table B-8.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Detroit Preparing At-Risk YouSh for Employment 





Coltttrt 1 


Cohort 1 


Cohort 






mi-n 


1990.9! 


1991»92 


Year 1 




Dropout otatus 


na 


na 


na 


+ 


+ 


na 


na 


Higher GPA 


na 


na 


na 


a;r 


R 


na 


na 


Higher number 
of credits 
earned 


na 


na 


na 






na 


na 


Fewer absences 


na 


na 


na 






na 


na 


Fewer coiurses 
failed 


na 


na 


na 






na 


na 



Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 
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Table 8-8.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Detroit Preparing At-Risk YouOt for Employment 



I "f"* «.P....A 


Cohort I 




Cofeort I 


Combined 








1990-91 


1991*9^ 


Vear I 




Higher GPA 


na 


na 


na 


+ 


+ 


na 


na 


Higher number 
of credits 
earned 


na 


na 


na 






na 


na 


Fewer absences 


na 


na 


na 






na 


na 


Fewer courses 
failed 


na 


na 


na 




+ 


na 


na 



Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-8.4 



Analysis of Project Dropout Rates, by Cohort by Year 
DetroU Preparing At-Risk Youth for Employment 



Year 


Colwrtl 








Trtattmnt 
Group 

(N« ) 


Control 
droBp 

(f^« ) 


Group 


Control 


■.■■tt 


■ % ' 


■:n ■ 




n 




n 1 


% 


1989-90 


na 


na 


na 


na 


na 


na 


na 


na 


1990-91 


na 


na 


na 


na 


3 


3% 


13 


14% 


1991-92 


na 


na 


na 


na 


10 


11% 


25 


26% 



Table B-8.5 



Analysis of School Affiliation and Student Perception Outcomes, by Cohort by Year 
Detroit Preparing At'Risk Youth for Employment 




"" II A . I .. Mn . I M if ; M^, ll v . W^. .. f l t l >>> ;..l 1jhU . 



mun tm^t tmm ymti vmt^ 



School thou^t safer 


na 


na 


na 






na 


na 


Teaching/Teachers better 


na 


na 


na 






na 


na 


Better job preparation 


na 


na 


na 






na 


na 


Counseling/Counselors better 


na 


na 


na 






na 


na 


More academic encouragement 


na 


na 


na 






na 


na 


Student perceptions 


Classmates should not 
misbehave 


na 


na 


na 






na 


na 


Future expectations are batter 


na 


na 


na 




-G 


na 


. na 


Many classmates are 
college bound 


na 


na 


na 






na 


na 



Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis of covaiiance), and R 
(reliability adjusted analysis of covaiiance). 
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Table B-8.6 



Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Detroit Preparing At-Risk Youth for Employment 











Cohorts 


€01 


Mill' IM>I.III<.II 














Yutr i 








- 


School thought safer 


na 


na 


na 






na 


na 


TeadidjagA'eachets better 


na 


na 


na 






na 


na 


Better job preparation 


na 


na 


na 






na 


na 


Counseling/Counselors better 


na 


na 


na 






na 


na 


More academic encouragement 


na 


na 


na 






na 


na 










Qassmates should not 
misbehave 


na 


na 


na 






na 


na 


Future expectations are better 


na 


na 


na 






Da 


na 


Many classmates are 
college bound 


na 


na 


na 






na 


na 



Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-9.1 



Characteristics of Study Participants 
Project COFFEE Vocational Training Program (Turtie Mountain) 



ClMur«ct«risti<^ : 




C«lMiH2 




Qrxxup 


Gtmp 


Group 
(N«ll) 


Coatrot 
Or««p 


Crrottp 




■ :; n ■ ■■ 


■■■■ 


n 


% 


It 


% 


II 


% 


n 




II 


% 






Female 


7 


44% 


8 


38% 


5 


45% 


9 


69% 


12 


44% 


17 


50% 


Male 


9 


56% 


13 


62% 


6 


55% 


4 


31% 


15 


56% 


17 


50% 


Race/Edmicity 




Native American 


16 


100% 


21 


100% 


11 


100% 


13 


100% 


27 


100% 


34 


100% 


Black 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


White 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Hispanic 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Asian/Pacific Islander 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 




Age appropriate 


2 


13% 


3 


14% 


2 


18% 


4 


31% 


4 


15% 


7 


21% 


Overage for grade 


14 


88% 


18 


86% 


9 


82% 


9 


69% 


23 


85% 


27 


79% 



Sample sizes for individual chatacteristics vary sli^tly across cfaaiacteristics and cohoits. 

/w /r. 



Table B-9.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Project COFFEE VocaHonal Training Program (Turtle Mountain) 



?::::::::S::^:::::::::%$:::::::^ 


iii^ii^Bi^Siiiii^^Kiiiii® 


Cohort 2 




— — — 


1990-91 


1991-92 


1990.91 


1991-92 


Year 1 




Dropout otatus 


+ 


+ 








+ 




Higher GPA 












G 




Higher number 
of credits 
earned 
















Fewer absences 
















Fewer courses 
failed 

















Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 
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Table 8-9.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Turtle Mountain) 



Outconie 


Cohort 1 


Coiioft t 


Combined 
Cohort 






tm-n 


i99(K9l 


1991»92 


Vear t 




Higher GPA 








+ 








Higher number 
of credits 
earned 
















Fewer absences 
















Fewer courses 
failed 








+ 









Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-9.4 



ERIC 



Analysis of Project Dropout Rates, by Cohort by Year 
Project COFFEE Vocational Training Program (Turtle Mountain) 



I 



Year 














Year 






Group : 
■ (N = 15) 


<3r<»ap 


■' ■■^Trtaiy*^ 


Control 
(N«13) 


TreatmcBt 
Group 


Co 
Gt 
<N 


vHanst 
foup. 


n 


















% 


iiliffii 


% 


1989-90 


3 


20% 


13 


68% 


na 


na 


na 


na 


Year 1 


5 


20% 


16 


50% 


1990-91 


5 


33% 


15 


79% 


2 


20% 


3 


23% 


Year 2 


10 


40% 


20 


63% 


1991-92 


10 


67% 


16 


84% 


5 


50% 


5 


38% 







23:^ 
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Table 8-9.5 



Analysis of School Affiliation and Student Perception Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Turtle Mountain) 





Cohort! 


Cohort 2 j 


















"''pimp^ \ 






Schod thought safer 












0,R 




Teaching/Teachers better 












na 


na 


Better job preparation 
















Counseling/Counselors better 












na 


na 


More academic eucouragement 
















SttUktnt jpetptpiiont 




Qassmates should not 
misbehave 
















Future expectations ate better 


-A,-R 










na 


. na 


Many classmates are 
college bound 

















Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis of covariance), aixi R 
(reliability adjusted analysis of covariance). 
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Table 8-9.6 

Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Turtle Mountain) 







1 

1 


CotMH2 


.llll)jMl1fl*lft 


1 m»s^ \ mm \ 




tmst 


mm 


YmtI 


















School ttxHight safer 
















Teadunj/Teacbeis better 












na 


na 


Better job preparation 
















Counseling/Counselors better 












na 


na 


More academic encouragement 
















ShidttUtimtPtUiwt 






Classmates should not 
misbehave 
















Future e:q)ectations arc better 












na 


na 


Many classmates are 
college bound 

















Note: Direction of statistical significance at p <05 is represented by +/-. 
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Table B-10.1 



Characteristics of Study Participants 
Project COFFEE Vocational Traitting Program (Fort Tottem) 



I 







1 ■^iM LL ] 1 1— n ri iPi^i 11 Ill 




Orwip 




Gr««p 


Crir«M|i 




' Y'^iiiiiiilrlittifif -a 


tt 


% 


at 


% 


II 








m 


% 


• 


% 1 








Female 


9 


39% 


4 


17% 


2 


13% 


26 


47% 


11 




30 


38% 


Male 


14 


61% 


20 


83% 


13 


87% 


29 


53% 


27 


71% 


49 


62% 


RaeelKAnkttt > "^K - ^ 


Native American 


23 


100% 


22 


92% 


14 


93% 


54 


98% 


37 


97% 


76 


96% 


Black 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


White 


0 


0% 


2 


8% 


1 


7% 


1 


2% 


1 


3% 


3 


4% 


Hispanic 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Asian/Pacific Islander 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Awf 


Age appropriate 


5 


22% 


8 


33% 


3 


20% 


18 


33% 


8 


21% 


26 


33% 


Overage for grade 


18 


78% 


16 


67% 


12 


80% 


37 


67% 


30 


79% 


53 


67% 



*S«npIe sizes for individual characteristics vary sligbdy across cfaaiacteiistics and cohorts. 
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Table B-10.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Totten) 





Cohort! 


Cohort t 


Cohort 

.Ill 








1990-91 




Vear 1 




Dropout Status 










+ 






Higher GPA 


R 














Higher number 
of credits 
earned 


-A,-R 










-G,-A 




Fewer absences 
















Fewer courses 
failed 


R 












-A,-R ' 



Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 
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Table B-10.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Totten) 





Cohort t 


Coboft t 






1990-91 


1991-92 




1991-92 


Vear 1 




Higher GPA 


+ 


+ 




+ 




+ 


+ 


Higher number 
of credits 
earned 
















Fewer absences 
















Fewer courses 
failed 


+ 


+ 













Note: Direction of statistical significance at p <.05 is represented by +/-. 



cod 

er|c ^-^^ 



Table B-10.4 

Analysis of Project Dropout Rates, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Totten) 










TreafxMfii^ 


Cootrot 
Gimp 


Tr«atawBt 
Group 


Control 




% 




iiili 




% 


B 




1989-90 


10 


43% 


3 


14% 


oa 


na 


na 


na 


1990-91 


15 


65% 


7 


32% 


1 


7% 


12 


28% 


1991-92 


17 


74% 


9 


41% 


1 


7% 


18 


42% 





1 nil... II II 1 illlllinllLllli 1 11 Jil iiiiiM.y nil 


Year 


Trtatakeitt 


'^Sf^m^ ■ 


s 






% 


Year 1 


11 


30% 


15 


23% 


Year 2 


16 


43% 


25 


38% 



2'U 



2-iO 
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Table B-10.5 



Analysis of School Affiliation and Student Perception Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Totten) 





C«bsrtl 
















iiiiiliiililiii^iililM 


School dioaght safer 
















Teacfaing/Teachen better 












na 


na 


Better job preparatioa 












-R 




Counseliog/Counselon better 


-A,-R 










na 


na 


More academic etKoaragement 
















StttdrntpttaptiQHS 


Qassinates should not 
misbehave 














A 


Future expectations are better 












na 


. na 


Many classmates are 
college bound 

















Note: Statistical significance at p <05 is represented by G (gain score analysis), A (analysis of covariance), and R 
(reliability adjusted analysis of covariance). 
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Table B-10.6 

Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Totten) 







1 C<>lltBft i 






""' 


1' i 






■ 
















School thought safer 
















Teadiing^eacbets better 












na 


na 


Better job preparation 




+ 












Counseling/Counselois better 












na 


na 


More academic encouragement 






















Classmates should not. 
misbehave 
















Future expectations are better 












na 


na 


Many classmates are 
college bound 

















Note: Direction of statistical significance at p <05 is represented by +/-. 
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Table B-11.1 

Characteristics of Study Participants 
Project COFFEE Vocational Training Program (Fort Berthold) 







Cohort 2 




Grmp 


Gr»op 




Cantrot 
Grovp 




Cwnpsrisoa 

(N s 3i> 




% 


tt 


% 


s 


% 


tt 




tt 




s 


% 


Gender 




Female 


5 


38% 


6 


35% 


5 


33% 


2 


14% 


10 


36% 


8 


26% 


Male 


8 


62% 


11 


65% 


10 


67% 


12 


86% 


18 


64% 


23 


74% 


Race/SOmkitf 


Native American 


13 


100% 


16 


94% 


14 


93% 


14 


100% 


27 


96% 


30 


97% 


Black 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


White 


0 


0% 


1 


6% 


1 


7% 


0 


0% 


1 


4% 


1 


3% 


Hispanic 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Asian/Paci&c Islander 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Age ' h 






Age appropriate 


4 


31% 


2 


13% 


4 


27% 


7 


50% 


8 


29% 


9 


30% 


Overage for grade 


9 


69% 


14 


88% 


11 


73% 


7 


50% 


20 


71% 


21 


70% 



^Sample sizes for individaal cbaracteristics vary slightly across chantcteiistics and cohorts. 



Table B-11.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Berthcld) 





Coltorli 


Cohort 2 


Cohort 


1989-00 1 1990-91 


1991*92 


199<k91 






t 


Dropout Status 
















Higher GPA 










G 






Higher number 
of credits 
earned 








-R 




-G,-A,-R 




Fewer absences 
















Fewer coxurses 
failed 










G 

1 




G 



Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 
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Table 8-11.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Berthold) 





iiiiSBlSiiilBBiW 

Cohort 1 


Cohoit 2 


Cotnbinea 
Mori 


Outcome 






mi.92 




1991.92 


Year 1 


|f';.";'|'N vi^ 


Higher GPA 
















Higher number 
of credits 
earned 








+ 


+ 


+ 




Fewer absences 
















Fewer courses 
failed 

















Note: Direction of statistical significance at p <.05 is represented by +/-. 
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Table B-1L4 



Analysis of Project Dropout Rates, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Berthold) 



w 



Year 


Cohort 1 






Y«ar 






Trtatmcot 

<Ntt 10) 


Control 


Trtatmeut 
Group 


Confrol 


(Nm24> 


Co 
<N 




:":[■■»':;■. 




tt 






% 




% 


tt 


% 




% 


1989-90 


3 


30% 


5 


33% 


na 


na 


Da 


na 


Year 1 


5 


21% 


6 


21% 


1990-91 


6 


60% 


6 


40% 


2 


14% 


1 


7% 


Year 2 


8 


33% 


9 


31% 


1991-92 


6 


60% 


6 


40% 


2 


14% 


3 


21% 
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Table B-ILS 



Analysis of School Affiliiation and Student Perception Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort BerthoU) 



J.I.J. .1 










im-n 








Y«arl 




















Scho<4 tiioa^ safer 
















Te«cfaiiig^eachen better 








-G,-A,-R 


-G,-A,-R 


na 


na 


Better job prepantion 


-A,.R 










-A,.R 




Couoseling/Counselois better 








-G,-A.-R 




na 


na 


More academic eocouragement 




















Qassmates should not 
misbehave 








A 




G,A^ 


G^ 


Futore expectatioas are better 


-A 










na 


na 


Many classmates are 
college bound 


R 















Note: Statistical significance at p <05 is represented by G (gain score analysis), A (analysis of covariance), and R 
(reliability adjusted analysis of covariance). 
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Table B-11.6 

Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort BerthoJd) 





1 














Yt«rl 




















School tbocght safer 
















Teacfaing^eacben better 












na 


na 


Better job prepandoa 










+ 






Couoseling/Counaelois better 












na 


na 


More academic eiKoaragement 






















Gassmates should not 
misbehave 














■ 


Patuie expectations are better 












na 


na 


Many classmates are 
college bound 


+ 


1 II 













Note: Direction of statistical significance at p <05 is represented by +/-. 
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Table B-12.1 



Characteristics of Study Participants 
Project COFFEE Vocational Training Program (Fort Yates) 







C«iiort2 






Tre* 

Qt 
(N* 


bmnt 




Tr«atai«nt 
<lr«ttp 


Caotrot 
Grenii 




CompvrisQii 




.■ ■»..:.■ 


% 


a 


% 




% 


tt 


% 


s 


* 


tt. 


% 


Gender 




Female 


8 


47% 


7 


44% 


5 


31% 


33 


53% 


13 


39% 


40 


51% 


Male 


9 


53% 


9 


56% 


11 


69% 


29 


47% 


20 


61% 


38 


49% 


RacefEtimieUy 








Native American 


16 


94% 


16 


100% 


14 


88% 


57 


92% 


30 


91% 


73 


94% 


Black 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


White 


1 


6% 


0 


0% 


2 


13% 


5 


8% 


3 


9% 


5 


6% 


Hispanic 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Asian/Pacific Islander 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


0 


0% 


Age 








Age appropriate 


4 


24% 


3 


19% 


6 


38% 


32 


52% 


10 


30% 


35 


45% 


Overage for grade 


13 


76% 


13 


81% 


10 


63% 


30 


48% 


23 


70% 


43 


55% 



♦Sample sizes for individual characteristics vary slightly across characteristics and cohorts. 



Table 8-12.2 



Analysis of School Performance Outcomes, by Cohort by Year 
Project COFFEE VocaOonal Training Program (Fort Yates) 



, ^ - 




Coboit t 


r.'. s^^^m m^^m 










1 dot .09 


Y i 




Dropout Status 
















Higher GPA 












GA 




— 

Higher number 

of credits 

earned 
















Fewer absences 








G 








Fewer courses 
failed 














-O 



Note: Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis 
of covariance), and R (reliability adjusted analysis of covariance). Dropout analysis is 
conducted with Mantel-Haenszel test. 
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Table B-12.3 



Gap Reduction Analysis of School Performance Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Yates) 





ColutrtI 


Cohort 2 


Mm 




1990.91 


1991-92 


1990*9] 


1991-92 






Higher GPA 


* 


* 


♦ ' 






na 


na 


Higher number 
of credits 
earned 


* 


* 


* 






na 


na 


Fewer absences 


* 


* 


* 






na 


na 


Fewer courses 
faUed 


« 


* 


* 






na 


na 



Note: Direction of statistical significance at p <.05 is represented by +/-. 
* Cohort without g^ reduction group 
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Table B-12.4 



Analysis of Project Dropout Rates, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Yates) 





Cohort 1 


Cotioril 




Ytar 






Ytar 


Treatooent 


Cotttroi 


TrcatiiMttt 
Group 


€<iBf)ro] 


Treatmeiot 
<5ronpt 


G« 

<N 


afrbt 


tt 


% 


tt 




tt 1 % 


» 1 % 


tt 








1989-90 


2 


13% 


1 


6% 


na 


na 


na 


na 


Year 1 


2 


6% 


13 


17% 


1990-91 


3 


19% 


4 


25% 


0 


0% 


12 


19% 


Year 2 


6 


19% 


18 


23% 


1991-92 


3 


19% 


5 


31% 


3 


19% 


14 


23% 
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Table 8-12.5 



Analysis of School Affiliation and Student Perception Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Yates) 



Owtcom* 




Cohort 2 


















School thought safer 




A 












Teaching/Teachers better 




-A,-R 








na 


na 1 


Better job preparation 
















Counseling/Counselors better 












na 


na 


More academic encouragement 




-A,-R 












S4ttd*nt ptratptioni 








Qassmates should not 
misbehave 
















Future erpectations are better 












na 


tia 


Many classmates are 
college bound 










R 




R 



Statistical significance at p <.05 is represented by G (gain score analysis), A (analysis of covariance). and R 
(rehabihty adjusted analysis of covariance). 
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Table B-12.6 



Gap Reduction Analysis of School Affiliation and Student Perception 
Outcomes, by Cohort by Year 
Project COFFEE Vocational Training Program (Fort Yates) 



j 






j 


ColKHrt2 1 


Cot 












im-n 1 








1 SehWflAifflaettm 




- - ^ s < 










School thought safer 


* 


* 


* 






na 


na 


Teachiiig/Tcachers better 


* 


* 


* 






na 


na 


Better job preparation 


* 


* 


* 






na 


na 


Counseling/Counselors better 


* 


* 


* 






na 


na 


More academic encouragement 


* 


* 








na 


na 


StudtHt fmtptiom 








Qassmates should not 
misbehave 


* 


* 








na 


na 


Future expectations are better 


* 


* 


* 






na 


na 


Many classmates are 
college botmd 


* 


* 


* 






na 


na 



Note: Direction of statistical significance at p <05 is represented by +/-. 
♦ Cohort without gap reduction group 
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Appendix C 
Derivation of Analytical Approarh 



The foUowing derivation is presented to explain the differences among the three 
analytical methods used in the dropout prevention study: analysis of covariance, reliabUity- 
adjusted analysis of covariance, and probabilistic gain scores. The basic paradigm is to 
assume that the data fit some reasonable model, in which a parameter represents the treatment 
effect, and to determine whether the results of specific computational models produce 
imbiased estimates of that parameter. 

The model for the data assumes that the post-treatment measure, Yj, and the pre- 
treatment measure, Y„ arc both distributed with error, E, and E„ and that the post-treatment 
measure includes an effect for the treatment, G, which is parameterized by Y- 
Suppose that the true model for a set of scores is represented by: 

Y„, = c, + T„ + E»,, Y^,, = c, + Tl + p T,j + Ej,, + Y G.,, 

for person j in group i at time 1 (pretest) and time 2 (posttest). The mean score at time 1 is 
c„ and the mean score at time 2 is c, plus a gain, Tl, for the control group, with an additional 
Y for the treatment group. The "trae" score, or the reliable variation in Y, is denoted by T. It 
can be assumed to have a mean of zero, or alternatively, the value of c, can be assumed to be 
zero. When the true score may be correlated with group membership, randomization or 
matching often is employed to assure that the correlation is as close to zero as possible. 
Finally, the gain may or may not be a function of the individual's original true score 
component (in which case p is different from one). The goal of the analysis is to obtain an 
unbiased estimate of Y- 



Gain Scores, Analysis of Covariance, and Rt ahilitv-Adiusted Analysis of Covariance 

The assumptions in this model are that the error temis, E, and E^, are uncorrelated 
with each other and with T and G. There is no assumption about the relative sizes of the 
variances of E, and Ej, although in the absence of evidence to the contrary one might expect 
them to be similar. If the within-group variance of the post-measure is larger than the 
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variance of the pre-measure, it may be an indication that (a) the variance of E, is larger than 
the variance of E„ (b) P > 1, or (c) the probabilistic gain score analysis is appropriate. 

The model as set forth is not idemifiable, and some additional data or an additional 
assumption is necessary to enable appropriate interpretation of analytical results. One 
possible simplifying assumption is that the variance of E, is zero. In that case, traditional 
analysis of covariance is appropriate. Unfortunately, in the present case, this assumption is 
patently false because the pre- and post-measures are the same scholastic perfomiance 
measures made in two different years and theie is little reason to believe that they were 
measured without error one year and with eixor the next. Nevertheless, simple analyses of 
covariance were performed as one of several methods of analysis used to "triangulate" the 
results. One rationale for this approach was that if there were substantial effects of a 
treatment, then they should show up across analytical methods. 

There is a second assumption that would validate the use of analysis of covariance, 
even if there were error in the pre-measurement: that there was no correlation between group 
membership and the true score at pre-test. If assignment to treatment and control conditions 
had been randomized, and if there were no differential attrition associated with values of the 
true score, then this would be a valid assumption. However, because there was substantial 
attrition in this smdy by its very nanire (i.e., a dropout prevention program evaluation), this 
assumption is questionable. 

A third assumption that can be made to render the model useable is that p is one. In 
other words, gains are not proportional to the true scores measured at pre-treatment. If gains 
are not restricted primarily to students who start out high or, alternatively, to smdems who 
start out low, then a simple gain score analysis, in which the pre measure is subtracted from 
the post-measure and the difference is the object of analysis, is appropriate. The primary 
indicator used to test for a difference between p and one is the variance of the scores (within- 
group). A larger variance at post-measurement may imply that (a) P is greater than one, 
(b) the variance of E, is larger than the variance of E„ or (c) the probabilistic gain score 
model is applicable. 

There is no simple way to differentiate these two possibilities, one of which caUs the 
simple gain score analysis into question while the other does not. However, there is an 
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additional process that may lead to larger post-measurement variances (within group) than 
pre-measurement variances. It may be that some individuals are engaged by the treatment (in 
either group) and therefore make a large gain while others make an "average" gain. This 
"probabilistic gain" model is identifiable, and it can account for a larger within-group variance 
at post-measurement than at pre-measurement. Analyses of the data using the probabilistic 
gain score model were carried out whenever (a) the variance at post-measurement was 
significantly larger than the pre-measurement variance and (b) a test of the appropriateness of 
the probabilistic gain score model was positive. The detaUs of the probabilisvlc gain score 
model are described at the end of this section. 

Finally, if information about the relative sizes of the variances of the true scores and 
the error scores at pre-measurement (i.e., about the reliability of is available, that 
information can be used in a "reliability-adjusted analysis of covariance." The problem for 
this approach in practice is that information about reliability is rarely known. The standard 
procedure in program evaluation, in which there is no extemal body of evidence about the 
reliability of measures, especially for the particular populations under study, is to use the 
pre/post correlation as the measure of reliability in carrying out reliability-adjusted analysis of 
covariance. 

In the present study, there were too fev/ scores at any particular site to estimate the 
"reliability" accurately. Indeed, at a few sites, the pre/post correlations of the observed scores 
were actually slightly negative. Therefore, the data were pooled across sites to obtain a single 
estimate of the reliability of each measure, the pooled-within-site, within-group, correlation 
between pre-measurement and post-measurement. 

Each of the possible analytical options will produce rmbiased estimates of the effects 
of the treatment if its assumptions are met, and each will produce biased estimates if its 
assumptions are violated. The following derivation shows the relations among these models 
and assumptions. To simplify the presentation, other covariates, such as race/ethnic group, 
are not included: the issue at hand concerns the treatment of pre-existing differences on the 
outcome measure, Y. 
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Consider three different linear regression models: 
Model 1: Y2 = Y, + G (simple analysis of covariance) 



Model 2: Y2 - Y, = G 

Model 3: Y^ - (b/rel) Y, = G 



(simple gain score analysis) 
(reliability-adjusted analysis of covariance) 



The first model generates estimates of effects both for Y„ which we shall refer to as 
P, and for G, which we shaU refer to as y. The second model generates only an estimate of y. 
The third model pre-estimates the relation between Y, and Yj (b) and inflates that estimate by 
dividing by the reliabUit>' estimate (ret) to eliminate the true-score component from the left- 
side of the model. Only the second and third of these models gives an unbiased estimate of y 
when there is a combination of pre-measurement error and correlation between group 
membership and true-score at pre-measurement. To see this, we have computed the expected 
value of the estimates of P and y, which we refer to as E(|5,Y). For simplicity in the 
presentation, we refer to cov(T,G) as and to o^j. / (<5^r + <y^Ei) as p. 

In Model 1, 
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cov(Kj,G) 
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from which we can compute the expected value of the estimate of y as: 

£,(Y) = ■ L 
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which clearly includes a bias, proportional to the covariance between the treatment and the 
pre-treatment measure and to the error variance in the pre-treatment measure. Likewise, the 
expected value of the estimate of P is: 
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In Model 2, 



E(Y) - 



cov((P-l)r+E2-E,+YG, G) 
E(^) = 



,2 



O G 



In other words, the expected value of the estimate of y is y if either p is one or the two 
groups are matched on the true score at pre-measurement. 

It should be noted that analysis of covariance (Model 1) can be viewed as a two-step 
rnalysis. In the fust step, the contribution of the pre-measurement to post-measurement is 
estimated; in the second step that contribution is eliminated by subtracting the appropriate 
multiple of the pre-measurement score from the post-measurement and then testing whether 
group membership accounts for significant variance of the difference. The appropriate 
multiplier to use in the subtraction is P, or an unbiased estimate of p. 

The problem is that if the pre-measurement has error, the regression coefficient 
obtained is not an unbiased estimate of p, as shown above (the expected value of the estimate 
of p is not p). In Model 3, that estimate of p is inflated by dividing by the "reliability," or 
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by cf J. / (sfj. + (f which approximately eliminates the bias in the estimate of p. (As 
shovm by the derivation for Model 1, it fails to eliminate the bias completely if X is non- 
zero.) This leads to the following equation: 

E(y) = — 



where rel is the "reliabiUty" of Y. Note that when the variances of Y, and Y^ are equal and 
the pre^st correlation is used as the reliability, thg value of rel is exactiy the estimate of p, 
so this is the same as the simple gain score model. 



cov((p— &-)r+£2-'Ei+YG, G) 
E(t) = 



and if the reliability inflation renders the estimate of beta equal to the parameter it is 
estimating, the first term wiH vanish. Thus the expected value oi fhe estimate of y is: 



which is biased to the extent that (a) there is a correlation between treatment group and tme 
score at pre-test and (b) the reliability adjustment is imprecise. 



Probabilistic Gain Score Analysis 

If the post-measurement variance is greater than the pre-measure variance, this may be 
because some individuals "gained" while others did not. The model presented in this section 
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addresses tiiis situation, providing an unbiased estimate of the treatment effect and a test for 
the appropriateness of this model. We suppose that there are two groups, i=l (Treatment) and 
i=2 (Control). There is a measure Y, taken on individuals both pre (Y „j) and post (Yj,j). This 
model retains the assumption, used in the simple gain score {uialysis, that the coefficient of T 
docs not change from pre-meas;irement to post-measurement. 

As before, the underlying model assumes that Y is measured with error at both points 
in time, with the same trae score variance component at both times. We suppose, further, 
that everybody may have some average gain, but that some individuals experience an 
additional gain. The nuU hypothesis is that the percentage of subjects that experience the 
additional gain is the same in group 1 and group 2. The alternative hypothesis is that more 
people receive gains in one of the groups. 

where 

jproiK5y=l)=7t+X and prob(,6y^l)^n 

T| represents the average gain for aU individuals, and Y is the additional gain for 
individuals who experienced the increase. 5 is the (1,0) variable indicating who got the 
additional gain; 7C is the base percentage of those who got the additional gain in both groups; 
and X, is the incremental number of individuals in the treatment group who got the gain. An 
implicit parameter, o, is the sum of variances of the two error terms. The statistic used for 
the test is the gain, G,j, the post minus pre difference for each individual. The test is a 
likelihood ratio test between two models, the one given and the one that assumes X, is zero. 

These equations can be solved for maximum likelihood estimators of the parameters in 
the following way. First, there are a discrete number of combinations of the delta i j's. The 
maximum likelihood is the maximum over these combinations of the maximum given the 
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combination of deltas. To maximize the likelihood with respect to pi, lambda, gamma, and 
eta, for a given set of deltas, we solve the equations that are of 'die fomi dlog(L)/d(tlieta)=0, 
where theta is some parameter. The log likelihood is given by: 

2 n, 

log(L)=-X;X; ^((0^,-11 -Y6J/o)2-(nj+n2)logov^ 
(5vlo«(" +^)+(l -fiJlogd -n -X) 

The equations can be solved easily, given a specific set of deltas. 

It is not essential to consider all possible values of the deltas separately, because, for a 
given average delta in each group, the allocation of the deltas among individual cases which 
maximizes the likelihood is clear: for example, given that four of eight deltas are 1 and four 
are 0, the maximum likelihood assignment is that the four I's go to the four individuals in the 
group with the highest gains.' 

One assumption of this model is that the deltas who received gains are uncorrected 
with the preexisting tme score differences, and therefore with Y„j. This correlation can be 
estimated by the correlation of "delta" with the score at pre-test; and this model can be used 
in place of the simple gain score analysis if (a) the variance of Y2,j is significantly greater 
than the variance of Y„j and (b) the correlation between Y„j with delta,j is not significantly 
different from zero. 



'sign unideatifiability can be removed by assuming that yis not negative. 
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